Dynamic flexible workshop multi-target online rescheduling method based on hierarchical MADDQN drive

Abstract

The invention provides a dynamic flexible workshop multi-target online rescheduling method based on hierarchical MADDQN driving, aiming at the problem of dynamic multi-target rescheduling in actual production. According to the method, two performance indexes, namely the minimum total tardiness and the maximum machine utilization rate, are optimized. The MADDQN comprises two DDQN agents, and the DDQN of the higher level is a controller for determining the temporary optimization target of the DDQN of the lower level. At each rescheduling point, the high-level DDQN considers a target which needs to emphasize optimization at present, the low-level DDQN considers process machine selection under the current optimization target, and the two intelligent agents cooperate with each other to carry out iterative training. Compared with other rules-based algorithms, the multi-target dynamic scheduling method has certain advantages in each performance index when processing a multi-target dynamic scheduling problem, and still has good performance when solving a large-scale problem.

Description

Technical Field

[0001] The present invention relates to flexible workshop scheduling in the field of intelligent manufacturing, and in particular to a multi-objective online rescheduling method for a dynamic flexible workshop based on hierarchical MADDQN drive. Background Art

[0002] As the manufacturing environment becomes increasingly complex, emergencies such as new order insertion and machine failures that constantly appear in dynamic scheduling scenarios have put forward higher requirements on traditional workshop scheduling systems. In the dynamic flexible job shop scheduling problem (FJSP-D), it is necessary not only to solve multi-objective optimization problems such as maximizing machine utilization and minimizing total delays, but also to be able to perform real-time scheduling according to environmental changes. In this context, multi-agent reinforcement learning methods are gradually applied to the optimization of dynamic scheduling, but they still face the problems of decision coordination and convergence under conflicting objectives. Therefore, an efficient, real-time and stable scheduling method is urgently needed to improve the adaptability and responsiveness of the production system. Summary of the invention

[0003] In view of the shortcomings of the prior art mentioned above, the present invention proposes an online rescheduling method based on a hierarchical multi-agent dual-depth Q network (MADDQN) for dynamic multi-objective scheduling problems in actual production. This method optimizes the two performance indicators of minimizing total delays and maximizing machine utilization. MADDQN includes two DDQN agents, and the higher-level DDQN is a controller that determines the temporary optimization target of the lower-level DDQN. At each rescheduling point, the high-level DDQN considers the current target that needs to be optimized, and the low-level DDQN considers the process machine selection under the current optimization target, and the two agents cooperate with each other for iterative training. Compared with other rule-based algorithms when dealing with multi-objective dynamic scheduling problems, the present invention has certain advantages in various performance indicators, and still has good performance when solving large-scale problems.

[0004] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0005] A multi-objective online rescheduling method for a dynamic flexible job shop based on hierarchical MADDQN driving is proposed. The hierarchical multi-agent dual-depth Q network algorithm is used to process the dynamic flexible job shop scheduling problem to achieve real-time optimization scheduling of multiple objectives. The method comprises the following steps:

[0006] Step 1: Modeling the multi-objective scheduling problem. In the dynamic FJSP problem of new order insertion, a two-stage multi-agent scheduling architecture model is constructed and multiple optimization objectives are set.

[0007] Step 2: Design the algorithm framework and hierarchical MADDQN framework, make decisions with hierarchical agent structure, and design state, action, and reward functions;

[0008] Step 3: A soft ε-greedy strategy is used to perform action selection, balancing exploration and exploitation. The two agents use experience replay and Huber loss function training to update their respective Q networks.

[0009] Step 4: Deploy the trained algorithm model in the workshop, and the intelligent agent selects appropriate scheduling rules to implement process task allocation and perform intelligent scheduling.

[0010] The two-stage multi-agent scheduling architecture model includes a high-level agent as a controller for selecting optimization targets according to the current environmental state, and a low-level agent as an executor for specific process selection and machine allocation.

[0011] The multiple optimization objectives include:

[0012] The objective function for minimizing total delay is:

[0013] The objective function to maximize the average machine utilization is:

[0014]

[0015] Among them, X i,j,k is the decision variable, t i,j,k It is process O ij Processing time on machine k, is a decision variable. If the process Assigned to machine M k On, then otherwise Ci,n i Indicates order J i The last step The completion time, D i It is order J i Deadline for delivery.

[0016] At each scheduling point, the high-level agent selects optimization objectives according to the current state, including: maximizing machine utilization when the machine load is high in the current environment, minimizing total delays when the order delivery pressure is high, and passing the optimization objectives to the low-level agents at the scheduling point; the low-level agents perform specific process selection and machine allocation at each scheduling point based on state feedback and optimization objectives based on the optimization objectives selected by the high-level agents, and generate operations.

[0017] The state characteristics of multi-agents include:

[0018] The state characteristics of the advanced agent include the total number of machines in the current workshop environment, the order arrival interval, and the order delivery relaxation factor;

[0019] The state characteristics of low-level agents include the average machine utilization Machine utilization standard deviation E std , Average process completion rate Average order completion rate Order completion rate standard deviation ACR j-std , estimated order delay rate, and actual order delay rate.

[0020] The action space includes:

[0021] The action space of the high-level agent includes:

[0022] Action 1 tends to optimize the performance indicator of minimizing total delay;

[0023] Action 2 tends to optimize the performance indicator of maximizing the average machine utilization;

[0024] The action space of the low-level agent includes: selecting the scheduling rule according to the current optimization goal and determining the specific allocation of machines for process tasks;

[0025] Action 1: Order selection: select the order with the minimum redundant time from the unfinished orders; Machine selection: select the machine using the minimum completion time allocation principle;

[0026] Action 2: Order selection: If there are delayed orders, select the order with the longest expected delay from the delayed orders for processing; if there are no delayed orders, select the order with the longest operation time from all unfinished orders for processing; Machine selection: Use the earliest available machine;

[0027] Action 3: Order selection: select the order with the largest estimated delay from all unfinished orders for processing; Machine selection: use the machine with the lowest utilization rate or the lowest workload among the available machines;

[0028] Action 4: Order selection: randomly select an order from the uncompleted orders; Machine selection: use the earliest available machine;

[0029] Action 5: Order selection: select the order with the largest estimated delay from all unfinished orders; Machine selection: use the earliest available machine.

[0030] Different optimization objectives correspond to different reward functions:

[0031] Under the optimization goal of minimizing delays, negative rewards are set according to the order delay time;

[0032] Under the optimization objective of maximizing machine utilization, positive rewards are given according to the reduction in machine idle rate.

[0033] The algorithm is trained by the following steps:

[0034] 1) Use the experience replay mechanism to store the state-action-reward sequence of each agent;

[0035] 2) Use the Huber loss function to update the Q-function network parameters to achieve joint training of high-level agents and low-level agents.

[0036] Intelligent scheduling of workshop orders is performed based on the trained algorithm model. The specific steps include:

[0037] Deploy the trained model to the workshop scheduling system;

[0038] The workshop scheduling system obtains the order status of all machines in the current workshop and gives production scheduling instructions based on the scheduling queue of the model;

[0039] After receiving the corresponding production scheduling instructions, the workshop sends them to specific machines to execute operations.

[0040] The production scheduling instructions include the process processing sequence and machine work allocation of each order, and are used to control several machines in the on-site workshop to execute the order process.

[0041] The present invention has the following beneficial effects and advantages:

[0042] 1. Through the hierarchical multi-agent architecture, the high-level agent dynamically selects the optimization goal, and the low-level agent implements the specific task allocation, so that the system can adaptively balance between minimizing the total delay and maximizing the machine utilization. This architecture improves the scheduling efficiency, significantly reduces the total delay and improves the machine utilization in a complex production environment.

[0043] 2. This method is highly adaptable in real time to adjust the scheduling scheme in dynamic scenarios such as new order insertion, machine failure, and priority change. The hierarchical MADDQN agent framework can achieve online rescheduling in a dynamic environment by acquiring and responding to the current production status in real time, thus reducing the negative impact of emergencies on production efficiency.

[0044] 3. The present invention adopts experience replay and Huber loss function to train Q network, combined with soft ε greedy strategy, which significantly improves the convergence speed and decision stability of scheduling algorithm. Experiments show that the method can still maintain efficient scheduling in large-scale production configuration, has good stability and convergence performance, and is suitable for a variety of complex production environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1This is a flow chart of the multi-objective online rescheduling method of a dynamic flexible workshop driven by hierarchical MADDQN according to the present invention.

[0046] Figure 2 This is the overall training framework diagram of MADDQN of the present invention.

[0047] Figure 3 Design a flow chart for the reward function of the present invention.

[0048] Figure 4 This is a flow chart of the algorithm training of the present invention. DETAILED DESCRIPTION

[0049] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0050] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The present invention is further described in detail below in conjunction with the accompanying drawings and examples.

[0051] The present invention aims to solve the problem of flexible job shop scheduling in the field of intelligent manufacturing, and specifically proposes a multi-objective optimization online rescheduling method applied to a dynamic environment, which realizes flexible job shop scheduling under order insertion based on a hierarchical multi-agent double deep Q network (Multi-Agent Double Deep Q Network, MADDQN). It includes the following steps:

[0052] Step 1: Modeling of multi-objective scheduling problems. A two-stage multi-agent scheduling mathematical model is constructed in the dynamic FJSP problem of new order insertion. Multiple optimization objectives are set. The objective function is to minimize the total delay and maximize the average utilization of the machine:

[0053]

[0054] The constraints are:

[0055]

[0056] Where n: order quantity; m: number of machines; Ci,n i Indicates order J i The last step The completion time of order J i Completion time; D i It is order J i The delivery deadline. ij Indicates order J i The jth process O ij The completion time, J i is the i-th order, O ij It is order J i The jth process. i,j,k is a decision variable. If process O ij Assigned to machine M k On, then X i,j,k =1, otherwise X i,j,k =0,X h,g,k is a decision variable. If process O hg Assigned to machine M k On, then X h,g,k =1, otherwise X h,g,k =0,M k is the kth machine, is a decision variable. If the process Assigned to machine M k On, then otherwise t i,j,k It is process O ij Processing time on machine k. i It is order J i The arrival time of h,g,k It is process O hg The processing time on machine k. The subscripts i,h are order indices. j,g is order J j and J g The process index, k is the machine index, C hg It is order J h The g-th process O hg Completion time. ijhg is a decision variable, used to represent process O ij and process O hg The relative processing priority between ijk Determines whether to operate O ij Assigned to machine k, and Y ijhg Determine any two operations O ij and O hg The relative processing priority between them.

[0057] Step 2: Design the algorithm framework and the hierarchical MADDQN framework, make decisions with a hierarchical agent structure, and design state, action, and reward functions.

[0058] Step 2-1: In the algorithm framework, two agents are set up in layers. The high-level agent (Higher DDQN) acts as a controller and selects optimization goals according to the current environment. When the machine load is high in the current environment, it chooses to maximize machine utilization; when the order delivery pressure is high, it chooses to minimize the total delay; the low-level agent (Lower DDQN) acts as an executor and performs specific task assignments according to the optimization goals set by the high-level agent. The overall training framework of MADDQN is as follows: Figure 2 shown.

[0059] Step 2-2: In order to improve the effectiveness and versatility of MADDQN, 10 general state features are extracted to comprehensively represent the production status of the rescheduling point, including 3 task-based features and 7 environment-based features.

[0060] The three basic characteristics are:

[0061] (1) The total number of machines in the workshop m;

[0062] (2) Order delivery slack time factor α i

[0063] (3) The time interval between the arrival of newly inserted orders follows the expected value λ of the exponential distribution.

[0064] The seven status characteristics are:

[0065] (1) Average machine utilization

[0066]

[0067] Where m is the number of machines, U k is the utilization of the kth machine.

[0068] (2) Standard deviation of machine utilization E std :

[0069]

[0070] (3) Average process completion rate

[0071]

[0072] Where n is the total number of orders, n i It is order J i The number of operations in OP ij It is order Ji The jth process O ij completion status.

[0073] (4) Average order completion rate

[0074]

[0075] Among them, ACR job [i] is the completion rate of the i-th order.

[0076] (5) Order completion rate standard deviation ACR j-std :

[0077]

[0078] (6) Estimated order delay rate:

[0079]

[0080] Where N is the number of operations that are estimated to be late, and N is the total number of unfinished operations late for all remaining orders.

[0082] (7) Actual order delay rate:

[0083]

[0084] Where N* is the number of operations that have actually been deferred.

[0085] late

[0086] Step 2-3: The action space of the two agents is, the action of the high-level agent is to select the current optimization goal, and the low-level agent chooses between machines and scheduling rules based on this goal.

[0087] High-level optimization objectives select the action space of the controller

[0088] If the controller selects action 1, it indicates that at this time it is more inclined to optimize the performance indicator of minimizing the total delay, which is recorded as g(t)=1; if the controller selects action 2, it indicates that at this time it is more inclined to optimize the performance indicator of maximizing the average machine utilization, which is recorded as g(t)=2.

[0089] The low-level allocation rule selects the action space of the actuator

[0090] Action 1:

[0091] Order selection: select the order with the minimum slack time from the unfulfilled orders;

[0092]

[0093] Among them, D i It is order J i Deadline for delivery, CT k (t) is the time at which machine m is rescheduled to point t. k The completion time of the last operation on the allocation.

[0094] Machine selection: Adopt the principle of minimum completion time allocation:

[0095]

[0096] in, It is the previous process The completion time, It is a process Processing time on machine k.

[0097] Action 2:

[0098] Order selection: If there are delayed orders, select the order with the longest expected delay from the delayed orders for processing; if there are no delayed orders, select the order with the longest operation time from all unfinished orders for processing:

[0099]

[0100] J i =

[0101] Among them, Tard job It's an order that has been delayed, Tard job is empty means there is no delayed order, UC job It is an unfulfilled order.

[0102] Select machines using the earliest available machine principle:

[0103]

[0104] Among them, M i(OP(t+1)) It means in the process Optional collection of machines.

[0105] Action 3:

[0106] Order selection: Select the order with the largest estimated delay from all unfinished orders for processing:

[0107]

[0108] Machine selection: Use the least utilized machine or the one with the lowest workload among the available machines for processing:

[0109]

[0110] Among them, r is an introduced random number, which is randomly generated between (0, 1). When selecting a machine, the machine with the lowest utilization rate is selected with a probability of 0.5, otherwise the machine with the lowest workload is selected.

[0111] Action 4:

[0112] Order selection: Randomly select an order from the outstanding orders.

[0113] Machine selection: Adopt the earliest available machine principle and directly select the earliest machine that can be processed.

[0114]

[0115] Action 5:

[0116] Order selection: Select the order with the largest estimated delay from all outstanding orders.

[0117]

[0118] Machine selection: Machine selection adopts the principle of the earliest available machine and directly selects the machine that can be processed earliest.

[0119]

[0120] Step 2-4: The reward function design is adjusted according to different optimization goals: when the optimization goal is to minimize the total delay, negative rewards are related to order delays; when the optimization goal is to maximize machine utilization, rewards are positively correlated with machine utilization. The flowchart of reward function design is as follows Figure 3 shown.

[0121] Step 3: Use a soft ε greedy strategy to perform action selection and balance exploration and exploitation. The two agents use experience replay and Huber loss function training to update their respective Q networks.

[0122] Step 3-1: During the training phase, both agents adopt a soft ε-greedy strategy to balance exploration and exploitation. In the initial training phase, the ε value is large, which makes the agents tend to explore different strategies. As training progresses, the ε value is gradually reduced to make more use of the known best strategy.

[0123] Step 3-2: Both high-level and low-level agents use the experience replay mechanism to randomly extract training samples from the experience pool, break the correlation between data, and improve the training stability of the model.

[0124] Step 3-3: Use the Huber loss function to update the Q network parameters of the two agents. Huber loss combines the advantages of mean square error and absolute error, ensuring that the gradient changes are more stable when encountering outliers, improving the convergence of the training process and the robustness of the model. The training flow chart of the algorithm is as follows: Figure 4 shown.

[0125] Step 4: The intelligent agent selects appropriate scheduling rules to implement process task allocation, and verifies the effectiveness of the proposed method through simulation.

[0126] Step 4-1: The agent dynamically selects appropriate scheduling rules based on the optimization goal to achieve efficient matching between processes and machines. Under the goal of minimizing delays, urgent processes are assigned to idle machines first; under the goal of maximizing machine utilization, tasks on low-load machines are filled first.

[0127] Step 4-2: The low-level agent performs specific machine allocation operations according to the optimization goal. The allocation strategy is dynamically adjusted according to the state feedback to ensure the adaptability of the production task under dynamic events such as order insertion and machine failure.

[0128] Step 4-3: Verify the applicability and real-time performance of the method in multi-objective optimization, and analyze various performance indicators to verify the effectiveness and applicability of the algorithm.

[0129] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A multi-objective online rescheduling method for dynamic flexible workshops based on hierarchical MADDQN driving, characterized in that: A dynamic flexible job shop scheduling problem is processed by a hierarchical multi-agent dual-depth Q network algorithm to achieve real-time optimization scheduling of multiple objectives. The method includes the following steps: Step 1: Modeling the multi-objective scheduling problem. In the dynamic FJSP problem of new order insertion, a two-stage multi-agent scheduling architecture model is constructed and multiple optimization objectives are set. Step 2: Design the algorithm framework and hierarchical MADDQN framework, make decisions with hierarchical agent structure, and design state, action, and reward functions; Step 3: A soft ε-greedy strategy is used to perform action selection, balancing exploration and exploitation. The two agents use experience replay and Huber loss function training to update their respective Q networks. Step 4: Deploy the trained algorithm model in the workshop, and the intelligent agent selects appropriate scheduling rules to implement process task allocation and perform intelligent scheduling.

2. According to claim 1, a multi-objective online rescheduling method for dynamic flexible workshops based on hierarchical MADDQN driving is characterized in that: The two-stage multi-agent scheduling architecture model includes a high-level agent as a controller for selecting optimization targets according to the current environmental state, and a low-level agent as an executor for specific process selection and machine allocation.

3. According to claim 1, a multi-objective online rescheduling method for dynamic flexible workshops based on hierarchical MADDQN driving is characterized in that: The multiple optimization objectives include: The objective function for minimizing total delay is: The objective function to maximize the average machine utilization is: Among them, X i,j,k is the decision variable, t i,j,k It is process O ij Processing time on machine k, is a decision variable. If the process Assigned to machine M k On, then otherwise Ci,n i Indicates order J i The last step The completion time, D i It is order J i Deadline for delivery.

4. A multi-objective online rescheduling method for dynamic flexible workshops based on hierarchical MADDQN drive according to claim 1 or 2, characterized in that: At each scheduling point, the high-level agent selects optimization objectives according to the current state, including: maximizing machine utilization when the machine load is high in the current environment, minimizing total delays when the order delivery pressure is high, and passing the optimization objectives to the low-level agents at the scheduling point; the low-level agents perform specific process selection and machine allocation at each scheduling point based on state feedback and optimization objectives based on the optimization objectives selected by the high-level agents, and generate operations.

5. According to claim 1, a multi-objective online rescheduling method for dynamic flexible workshops based on hierarchical MADDQN driving is characterized in that: The state characteristics of multi-agents include: The state characteristics of the advanced agent include the total number of machines in the current workshop environment, the order arrival interval, and the order delivery relaxation factor; The state characteristics of low-level agents include the average machine utilization Machine utilization standard deviation E std , Average process completion rate Average order completion rate Order completion rate standard deviation ACR j-std , estimated order delay rate, and actual order delay rate.

6. According to claim 1, a multi-objective online rescheduling method for dynamic flexible workshops based on hierarchical MADDQN driving is characterized in that: The action space includes: The action space of the high-level agent includes: Action 1 tends to optimize the performance indicator of minimizing total delay; Action 2 tends to optimize the performance indicator of maximizing the average machine utilization; The action space of the low-level agent includes: selecting the scheduling rule according to the current optimization goal and determining the specific allocation of machines for process tasks; Action 1: Order selection: select the order with the minimum redundant time from the unfinished orders; Machine selection: select the machine using the minimum completion time allocation principle; Action 2: Order selection: If there are delayed orders, select the order with the longest expected delay from the delayed orders for processing; if there are no delayed orders, select the order with the longest operation time from all unfinished orders for processing; Machine selection: Use the earliest available machine; Action 3: Order selection: select the order with the largest estimated delay from all unfinished orders for processing; Machine selection: use the machine with the lowest utilization rate or the lowest workload among the available machines; Action 4: Order selection: randomly select an order from the uncompleted orders; Machine selection: use the earliest available machine; Action 5: Order selection: select the order with the largest estimated delay from all unfinished orders; Machine selection: use the earliest available machine.

7. According to claim 1, a multi-objective online rescheduling method for dynamic flexible workshops based on hierarchical MADDQN driving is characterized in that: Different optimization objectives correspond to different reward functions: Under the optimization goal of minimizing delays, negative rewards are set according to the order delay time; Under the optimization objective of maximizing machine utilization, positive rewards are given according to the reduction in machine idle rate.

8. According to claim 1, a multi-objective online rescheduling method for dynamic flexible workshops based on hierarchical MADDQN driving is characterized in that: The algorithm is trained by the following steps: 1) Use the experience replay mechanism to store the state-action-reward sequence of each agent; 2) Use the Huber loss function to update the Q-function network parameters to achieve joint training of high-level agents and low-level agents.

9. According to the multi-objective online rescheduling method of a dynamic flexible workshop based on hierarchical MADDQN drive as described in claim 1, it is characterized in that: Intelligent scheduling of workshop orders is performed based on the trained algorithm model. The specific steps include: Deploy the trained model to the workshop scheduling system; The workshop scheduling system obtains the order status of all machines in the current workshop and gives production scheduling instructions based on the scheduling queue of the model; After receiving the corresponding production scheduling instructions, the workshop sends them to specific machines to execute operations.

10. The multi-objective online rescheduling method for dynamic flexible workshops based on hierarchical MADDQN drive according to claim 8, characterized in that: The production scheduling instructions include the process processing sequence and machine work allocation of each order, and are used to control several machines in the on-site workshop to execute the order process.

Images