Dynamic flexible workshop scheduling method based on event-driven DDQN algorithm

Abstract

The invention discloses a dynamic flexible workshop scheduling method based on an event-driven DDQN algorithm. The method is used for solving the problem of flexible order workshop dynamic scheduling under the condition that a new order is inserted. The method comprises the following steps: firstly, considering the condition of random insertion of a new order, establishing a dynamic FJSP mathematical model taking minimization of total tardiness as an optimization target, converting the dynamic FJSP mathematical model into a multi-stage decision process based on event driving, and carrying out Markov decision process modeling of a scheduling problem; then seven key state characteristics such as machine utilization rate are designed, five composite scheduling rules are designed to form an action space, and a reward function is designed according to the variable quantity of part of the state characteristics; in addition, a soft epsilon greedy behavior strategy is designed according to the problem scale, and exploration and utilization are reasonably balanced. Through simulation research on a plurality of production scheduling examples, the result shows that the method can effectively cope with a dynamic event of random insertion of a new order, the total delay time is remarkably shortened, and good performance is shown in the aspects of convergence speed and solving efficiency.

Description

Technical Field

[0001] The invention belongs to the field of manufacturing production scheduling, in particular to a dynamic flexible workshop scheduling method based on an event-driven DDQN algorithm. Background Art

[0002] Traditional flexible order-based shop scheduling is usually designed for static environments. However, in actual production processes, dynamic events such as random insertion of new orders occur frequently, which can cause the pre-made scheduling scheme to fail, thus affecting production efficiency. Existing research mostly focuses on static FJSP solutions, but rarely considers how to quickly respond to changes in dynamic environments and achieve efficient real-time scheduling. Therefore, it is particularly important to develop a scheduling method that can adapt to dynamic changes and flexibly adjust production plans. Summary of the invention

[0003] In view of the shortcomings of the prior art described above, the purpose of the present invention is to propose a dynamic flexible shop scheduling method based on an event-driven DDQN algorithm for the dynamic FJSP single-objective optimization problem. First, considering the situation of random insertion of new orders, a dynamic FJSP mathematical model with minimizing the total delay as the optimization goal is established, which is transformed into a multi-stage decision-making process based on event-driven, and the Markov decision process modeling of the scheduling problem is carried out. Then, seven key state characteristics such as machine utilization rate are designed, and five composite scheduling rules are designed to form the action space, and the reward function is designed according to the change amount of some state characteristics. In addition, a soft ε greedy behavior strategy is designed according to the problem scale, which reasonably balances exploration and utilization. After simulation studies on multiple production scheduling examples, the results show that the proposed method can effectively deal with the dynamic event of random insertion of new orders, significantly reduce the total delay time, and show good performance in terms of convergence speed and solution efficiency.

[0004] The technical solution adopted by the present invention to achieve the above-mentioned purpose is: a dynamic flexible workshop scheduling method based on an event-driven DDQN algorithm, comprising the following steps:

[0005] S1. Establish a mathematical model for dynamic flexible order-shop scheduling and define the scheduling problem as a Markov decision process (MDP), including state space, action space and reward function.

[0006] S2. Through the dual-depth Q network algorithm combined with the reinforcement learning mechanism, the Markov decision process is solved according to the current dynamics, and the scheduling queue is dynamically adjusted; through the soft ε greedy strategy, a balance is achieved between trying new scheduling strategies and executing existing strategies to optimize the scheduling queue;

[0007] S3, when a dynamic event occurs, step S2 is triggered to reschedule and update the scheduling queue;

[0008] S4. Deploy the trained algorithm model on the workshop orders for intelligent scheduling.

[0009] The state space of the scheduling problem includes: average machine utilization Machine utilization standard deviation E std , Average process completion rate Average order completion rate Order completion rate standard deviation ACR j-std , ELR job , Order Actual Delay Rate ALR job .

[0010] The action space of the scheduling problem includes:

[0011] Action 1: Select order J with the minimum redundancy time from the unfinished orders i ; Select machine m using the minimum completion time allocation principle k ;

[0012] Action 2: If there are delayed orders, select the order with the longest expected delay from the delayed orders J i ; If there are no delayed orders, select the order with the longest operation time from all unfinished orders J i ; Select machine m using the earliest available machine principle k :

[0013] Action 3: Select the order J with the largest estimated delay from all unfinished orders i ; Use the least utilized machine or the one with the lowest workload among the available machines k :

[0014] Action 4: Randomly select an order J from the uncompleted orders i ; Select machine m using the earliest available machine principle k ;

[0015] Action 5: Select order J with the largest estimated delay from all unfinished orders i ; Select machine m using the earliest available machine principle k .

[0016] By continuously considering the current state s t and the next state s t+1 The actual order delay rate and the average machine utilization rate in the state-action pair (s t ,a t ) t ,include:

[0017] When the actual delay rate of the next state is lower than the actual delay rate of the current state or the delay rates are the same and the machine utilization rate increases, a positive reward is given to the agent;

[0018] When the delay rate is the same and the machine utilization rate does not decrease much, no reward is given to the agent;

[0019] When the actual delay rate becomes higher or other situations occur, a negative reward is given to the agent.

[0020] The dual-depth Q network algorithm is used for learning and optimization through the following steps:

[0021] Set up a reasonable experience replay mechanism, store the state-action-reward mapping relationship established by the agent in the production process in the experience pool, and train it through random sampling to obtain the reward target value y j ; According to the reward target value y j Map to the corresponding action, and obtain multiple candidate action scheduling queues after multiple time steps are linked; then calculate the optimization target of each action scheduling queue, and determine the optimal action of the current time step according to the optimization target;

[0022] During the training process, the target network is used to stabilize the Q-value update, and the online network parameters are periodically copied to the target network to reduce policy fluctuations and improve model convergence.

[0023] The design of the soft ε-greedy strategy achieves a balance between exploring new scheduling strategies and utilizing the current best scheduling solution by dynamically adjusting the ε value, including:

[0024] In the initial stage, the ε value is large and the system tends to explore new scheduling strategies;

[0025] As the training process progresses, the ε value is gradually reduced so that the system makes more use of the currently known best scheduling strategy to improve scheduling efficiency.

[0026] The optimization goal is to minimize the total delay, and its mathematical expression is:

[0027]

[0028] Among them, Ci,n i Indicates order J i The last step The completion time of order J i completion time.

[0029] The event-driven mechanism is used to handle various dynamic events that occur during the production process, including the insertion of high-priority orders, equipment failures, and material shortages.

[0030] The specific steps of intelligently scheduling the workshop orders according to the trained algorithm model include:

[0031] Deploy the trained model to the workshop scheduling system;

[0032] The workshop scheduling system obtains the order status of all machines in the current workshop and gives production scheduling instructions based on the scheduling queue of the model;

[0033] After receiving the corresponding production scheduling instructions, the workshop sends them to specific machines to execute operations.

[0034] The production scheduling instructions include the process processing sequence and machine work allocation of each order, and are used to control several machines in the on-site workshop to execute the order process.

[0035] The present invention has the following beneficial effects and advantages:

[0036] 1. The present invention can respond to dynamic changes in the production process in real time, such as the insertion of new orders, and provide a flexible and efficient scheduling solution, which effectively improves the real-time response capability of the equipment in the production scheduling process, improves production efficiency, reduces the total delay time, and increases the on-time delivery rate of orders, which can effectively improve the competitiveness of the enterprise.

[0037] 2. The present invention adopts a more intelligent decision-making algorithm. By using an improved dual deep Q network (DDQN) algorithm, the present invention improves the intelligence level of scheduling decisions, reduces dependence on manual experience, and makes the scheduling process more automated and accurate.

[0038] 3. The present invention achieves an effective balance between exploring new strategies and using known strategies, and designs a soft ε greedy behavior strategy, which helps the algorithm to adapt to new production environments while maintaining good convergence. Moreover, the present invention is not only applicable to trained production configurations, but also to new environments that have not been trained, showing good generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A flow chart is designed for the dynamic flexible workshop scheduling method based on the event-driven DDQN algorithm of the present invention.

[0040] Figure 2 A schematic diagram of the general process of solving shop orders using reinforcement learning of the present invention.

[0041] Figure 3 This is a flow chart of the event-driven process scheduling of the present invention. DETAILED DESCRIPTION

[0042] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the specific implementation method of the present invention is described in detail below in conjunction with the accompanying drawings. In the following description, many specific details are set forth to facilitate a full understanding of the present invention. However, the present invention can be implemented in many other ways different from those described herein, and those skilled in the art can make similar improvements without violating the connotation of the invention, so the present invention is not limited by the specific implementation disclosed below.

[0043] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art of the present invention. The terms used in the specification of the invention herein are only for the purpose of describing specific embodiments and are not intended to limit the present invention. The present invention is further described in detail below in conjunction with the accompanying drawings and examples.

[0044] The present invention provides a method for solving the problem of dynamic scheduling of a flexible job shop (FJSP) when new orders are inserted. The method combines deep reinforcement learning with traditional scheduling rules, aims to improve production efficiency and reduce total delays by optimizing scheduling strategies, and includes the following steps:

[0045] Step 1: Establish a mathematical model for dynamic flexible order-shop scheduling and define the scheduling problem as a Markov decision process (MDP), including the design of state space, action space, and reward function;

[0046] Step 1-1, the FJSP problem has dynamic interference factors such as random order insertion. In the actual production process, the time interval between newly inserted orders follows an exponential distribution, that is, at any given time point, the probability of the next order arriving is independent of the previous arrival time. Assume that T is a continuous event variable, representing the event interval between two new order insertions, and the probability density function of T is:

[0047]

[0048] Step 1-2, to ensure that the algorithm is more widely adaptable and shows good performance indicators under different urgency levels, set the relaxation factor α of the order i , whose range is (0,2). The smaller the slack factor of an order, the shorter the idle time between the order arrival date and the deadline, which indicates that the order has a higher priority and is more urgent than other orders. Otherwise, it indicates that the order has a lower priority.

[0049] In steps 1-3, the optimization goal of the FJSP problem is to minimize the total delay, and its mathematical expression is:

[0050]

[0051] Where n is the number of orders, Ci,n i Indicates order J i The last step The completion time of order J i Completion time; D i It is order J i Deadline for delivery.

[0052] The constraints are:

[0053]

[0054] Among them, C ij Indicates order J i The jth process O ij The completion time, J i is the i-th order, O ij It is order J i The jth process. i,j,k is a decision variable. If process O ij Assigned to machine M k On, then X i,j,k =1, otherwise X i,j,k =0,X h,g,k is a decision variable. If process O hg Assigned to machine M k On, then X h,g,k =1, otherwise X h,g,k =0,M k is the kth machine. i,j,k It is process O ij The processing time on machine k. i It is order J i The arrival time of h,g,k It is process O hg The processing time on machine k. The subscripts i,h are order indices. j,g is order J j and J g The process index, k is the machine index, C hg It is order J h The g-th process O hg Completion time. ijhg is a decision variable, used to represent process O ij and process O hg The relative processing priority between ijk Determines whether to operate O ij Assigned to machine k, and Y ijhg Determine any two operations Oij and O hg The relative processing priority between them.

[0055] In steps 1-4, the seven characteristics of the system's state space are:

[0056] (1) Average machine utilization

[0057]

[0058] Where m is the number of machines, U k is the utilization of the kth machine.

[0059] (2) Standard deviation of machine utilization rate E std :

[0060]

[0061] (3) Average process completion rate

[0062]

[0063] Where n is the total number of orders, n i It is order J i The number of operations in OP ij It is order J i The jth process O ij completion status.

[0064] (4) Average order completion rate

[0065]

[0066] Among them, ACR job [j] is the completion rate of the i-th order.

[0067] (5) Order completion rate standard deviation ACR j-std :

[0068]

[0069] (6) Estimated order delay rate:

[0070]

[0071] Among them, N late is the number of operations that are expected to be late, N total is the number of outstanding operations for all remaining orders.

[0072] (7) Actual order delay rate:

[0073]

[0074] Among them, N late * is the number of operations that have actually been deferred.

[0075] Steps 1-5, the five rules of the system's action space are:

[0076] Action 1:

[0077] Order selection: select the order with the minimum slack time from the unfulfilled orders;

[0078]

[0079] Among them, D i It is order J i Deadline for delivery, CT k (t) is the time at which machine m is rescheduled to point t. k The completion time of the last operation on the allocation.

[0080] Machine selection: Adopt the principle of minimum completion time allocation:

[0081]

[0082] in, It is the previous process The completion time, It is a process Processing time on machine k.

[0083] Action 2:

[0084] Order selection: If there are delayed orders, select the order with the longest expected delay from the delayed orders for processing; if there are no delayed orders, select the order with the longest operation time from all unfinished orders for processing:

[0085]

[0086] J i =

[0087] Among them, Tard job It's an order that has been delayed, Tard job is empty means there is no delayed order, UC job It is an unfulfilled order.

[0088] Select machines using the earliest available machine principle:

[0089]

[0090] Among them, Mi(OP(t+1)) It means in the process Optional collection of machines.

[0091] Action 3:

[0092] Order selection: Select the order with the largest estimated delay from all unfinished orders for processing:

[0093]

[0094] Machine selection: Use the least utilized machine or the one with the lowest workload among the available machines for processing:

[0095]

[0096] Among them, r is an introduced random number, which is randomly generated between (0, 1). When selecting a machine, the machine with the lowest utilization rate is selected with a probability of 0.5, otherwise the machine with the lowest workload is selected.

[0097] Action 4:

[0098] Order selection: Randomly select an order from the outstanding orders.

[0099] Machine selection: Adopt the earliest available machine principle and directly select the earliest machine that can be processed.

[0100]

[0101] Action 5:

[0102] Order selection: Select the order with the largest estimated delay from all outstanding orders.

[0103]

[0104] Machine selection: Machine selection adopts the principle of the earliest available machine and directly selects the machine that can be processed earliest.

[0105]

[0106] Steps 1-6, by continuously considering the current state s t and the next state s t+1 The state-action pair (s t ,a t ) tWhen the actual delay rate of the next state is lower than the actual delay rate of the current state or the delay rates are the same and the machine utilization rate increases, a positive reward is given to the agent, which helps the agent learn more effective scheduling strategies to reduce the delay rate and improve the machine utilization rate; when the delay rates are the same and the machine utilization rate does not decrease much, no reward is given to the agent, which means that the system remains relatively stable in this case and does not require obvious intervention by the agent; when the actual delay rate increases or other situations, a negative reward is given to the agent, which helps the agent avoid taking strategies that lead to increased delay rates or other unfavorable situations.

[0107]

[0108] Among them, ALR job (t+1) represents the actual delay rate of the order at time t+1, ALR job (t) represents the actual delay rate of the order at time t; U ave (t+1) is the average machine utilization at time t+1, U ave (t) is the average machine utilization at time t.

[0109] Step 2: Through the dual deep Q network (DDQN) algorithm, combined with the reinforcement learning mechanism, the scheduling decision is dynamically adjusted according to the current production status such as machine utilization, order queue status, etc. The general process of solving the order workshop scheduling by reinforcement learning is as follows: Figure 2 shown.

[0110] Step 2-1, the DDQN algorithm is a further extension of the DQN algorithm. First, perform the initialization operation: Initialize the capacity of the experience replay pool D to N. Initialize the minimum batch size k and the learning rate η (step length). Randomly initialize the online Q network weight θ. (4) Set the weight θ of the target Q network to the same initial value as the online Q network.

[0111] Step 2-2, then do circuit training:

[0112] (1) Reset the environment and clear the current scheduling plan and scheduling results to obtain the initial state s1. The initial state contains information such as the machine status, standard deviation, and actual order delay rate, and all these features are set to 0 at the beginning.

[0113] (2) For each time step t from 1 to T:

[0114] Choose an action a according to the soft ε greedy strategy t , the specific form of the soft ε greedy strategy is:

[0115]

[0116] Where Pnum is the total number of operations and μ is a control parameter.

[0117] If no exploration is performed, then the choice is to make Q(s t ,a;θ) the maximum action a t .

[0118] Perform the selected actiona t , select the order and machine according to the selected rules, and then calculate the instant reward r t And observe the next state s t+1 .

[0119] Update production status, set new status t+1 Contains updated machine status and actual delinquency rate.

[0120] The transfer quadruple (s t ,a t ,r t ,s t+1 ) is stored in the experience replay pool D.

[0121] Randomly draw a batch of samples (s j ,a j ,r j ,s j+1 ).

[0122] Calculate the reward target value y j : If the episode terminates at step t, then y j =r j .

[0123] otherwise,

[0124]

[0125] Here γ is the discount factor, the first Q function is used to select the optimal action, and the second Q function uses the target network to evaluate the value of this action.

[0126] Use gradient descent or similar methods to update the weights θ of the online Q network to minimize the loss function:

[0127]

[0128] The weight θ of the target Q network is updated to the weight θ of the online Q network every certain number of iterations or regularly.

[0129] Among them, the reward target value y j It is used to find the corresponding action, and obtain the action scheduling queue after multiple steps of linking. Then calculate the total delay rate of each action scheduling queue, and determine the most important action at the moment based on the principle of minimum total delay rate.

[0130] Among them, the soft ε greedy strategy strikes a balance between trying new scheduling strategies and executing known effective strategies to optimize the scheduling effect; ε is a linear annealing parameter, and its value gradually decreases during the training process, that is, more emphasis is placed on exploration in the initial stage, and more emphasis is placed on utilizing known information as time goes by.

[0131] according to Calculate the soft ε greedy behavior strategy, total number of operations OP num The bigger, The smaller it is, the slower the exploration rate ε decreases, thereby strengthening exploration and weakening utilization.

[0132] Step 3: Use the event-driven mechanism to trigger rescheduling when dynamic events occur, including the insertion of high-priority orders, equipment failure, material loss, etc. The scheduling flow chart of the event-driven mechanism is as follows: Figure 3 shown.

[0133] (1) For the new order insertion problem, after receiving a new order at any time point, the system will adjust the current scheduling queue in real time based on the DDQN algorithm to ensure that the process of the new order can be reasonably inserted into the existing plan;

[0134] (2) For machine failure, when a production device fails, the system reallocates pending tasks to other available machines based on the current task load and repair time of the failed machine;

[0135] (3) For process delay issues: If the estimated completion time of a process is delayed, the system will automatically trigger rescheduling to prevent subsequent processes from affecting the overall progress due to delays in the previous processes.

[0136] (4) Regarding the priority change problem, when customers change their order requirements (such as order quantity, specifications, and latest completion time), the scheduling strategy can quickly respond to the priority changes and ensure that the production plan is completed on time.

[0137] The above description is only a preferred embodiment of the present invention and does not limit the present invention in any way. Any simple modification, change and equivalent structural change made to the above embodiment based on the technical essence of the present invention still falls within the protection scope of the technical solution of the present invention.

Claims

1. A dynamic flexible workshop scheduling method based on event-driven DDQN algorithm, characterized in that: The following steps are involved: S1. Establish a mathematical model for dynamic flexible order-shop scheduling and define the scheduling problem as a Markov decision process (MDP), including state space, action space and reward function. S2. Through the dual-depth Q network algorithm combined with the reinforcement learning mechanism, the Markov decision process is solved according to the current dynamics, and the scheduling queue is dynamically adjusted; through the soft ε greedy strategy, a balance is achieved between trying new scheduling strategies and executing existing strategies to optimize the scheduling queue; S3, when a dynamic event occurs, step S2 is triggered to reschedule and update the scheduling queue; S4. Deploy the trained algorithm model on the workshop orders for intelligent scheduling.

2. A dynamic flexible workshop scheduling method based on event-driven DDQN algorithm according to claim 1, characterized in that: The state space of the scheduling problem includes: average machine utilization Machine utilization standard deviation E std , Average process completion rate Average order completion rate Order completion rate standard deviation ACR j-std , ELR job , actual order delay rate ALR.

3. The dynamic flexible workshop scheduling method based on the event-driven DDQN algorithm according to claim 1 is characterized in that: The action space of the scheduling problem includes: Action 1: Select order J with the minimum redundancy time from the unfinished orders i ; Select machine m using the minimum completion time allocation principle k ; Action 2: If there are delayed orders, select the order with the longest expected delay from the delayed orders J i ; If there are no delayed orders, select the order with the longest operation time from all unfinished orders J i ; Select machine m using the earliest available machine principle k : Action 3: Select the order J with the largest estimated delay from all unfinished orders i ; Use the least utilized machine or the one with the lowest workload among the available machines k : Action 4: Randomly select an order J from the uncompleted orders i ; Select machine m using the earliest available machine principle k ; Action 5: Select order J with the largest estimated delay from all unfinished orders i ; Select machine m using the earliest available machine principle k .

4. The method for dynamic flexible workshop scheduling based on event-driven DDQN algorithm according to claim 1, characterized in that: By continuously considering the current state s t and the next state s t+1 The actual order delay rate and the average machine utilization rate in the state-action pair (s t ,a t ) t ,include: When the actual delay rate of the next state is lower than the actual delay rate of the current state or the delay rates are the same and the machine utilization rate increases, a positive reward is given to the agent; When the delay rate is the same and the machine utilization rate does not decrease much, no reward is given to the agent; When the actual delay rate becomes higher or other situations occur, a negative reward is given to the agent.

5. The method for dynamic flexible workshop scheduling based on event-driven DDQN algorithm according to claim 1, characterized in that: The dual-depth Q network algorithm is used for learning and optimization through the following steps: Set up a reasonable experience replay mechanism, store the state-action-reward mapping relationship established by the agent in the production process in the experience pool, and train it through random sampling to obtain the reward target value y j ; According to the reward target value y j Map to the corresponding action, and obtain multiple candidate action scheduling queues after multiple time steps are linked; then calculate the optimization target of each action scheduling queue, and determine the optimal action of the current time step according to the optimization target; During the training process, the target network is used to stabilize the Q-value update, and the online network parameters are periodically copied to the target network to reduce policy fluctuations and improve model convergence.

6. The method for dynamic flexible workshop scheduling based on event-driven DDQN algorithm according to claim 1, characterized in that: The design of the soft ε-greedy strategy achieves a balance between exploring new scheduling strategies and utilizing the current best scheduling solution by dynamically adjusting the ε value, including: In the initial stage, the ε value is large and the system tends to explore new scheduling strategies; As the training process progresses, the ε value is gradually reduced so that the system makes more use of the currently known best scheduling strategy to improve scheduling efficiency.

7. A dynamic flexible workshop scheduling method based on event-driven DDQN algorithm according to claim 5, characterized in that: The optimization goal is to minimize the total delay, and its mathematical expression is: Among them, Ci,n i Indicates order J i The last step The completion time of order J i completion time.

8. The method for dynamic flexible workshop scheduling based on event-driven DDQN algorithm according to claim 1, characterized in that: The event-driven mechanism is used to handle various dynamic events that occur during the production process, including the insertion of high-priority orders, equipment failures, and material shortages.

9. A dynamic flexible workshop scheduling method based on event-driven DDQN algorithm according to claim 1, characterized in that: The specific steps of intelligently scheduling the workshop orders according to the trained algorithm model include: Deploy the trained model to the workshop scheduling system; The workshop scheduling system obtains the order status of all machines in the current workshop and gives production scheduling instructions based on the scheduling queue of the model; After receiving the corresponding production scheduling instructions, the workshop sends them to specific machines to execute operations.

10. The method for dynamic scheduling of order workshops considering new energy consumption according to claim 9 is characterized in that: The production scheduling instructions include the process processing sequence and machine work allocation of each order, and are used to control several machines in the on-site workshop to execute the order process.

Images