Dynamic flexible job shop multi-target scheduling method based on deep reinforcement learning

Abstract

The invention provides a dynamic flexible job-shop multi-target scheduling method based on deep reinforcement learning. The dynamic flexible job-shop multi-target scheduling method comprises the following steps: converting a deterministic scheduling problem into a Markov decision model; based on a Markov decision model, constructing a deterministic scheduling algorithm based on MOHDQN, designing a training process, training the algorithm, and testing on a random example to form a deterministic scheduling model agent; constructing a Markov jump decision process, and converting the event-driven pre-reactive scheduling problem into the Markov jump decision process; based on an HDRQN algorithm, constructing a training process, and interactively training an event-driven pre-reactive scheduling agent with a simulation environment; and testing an interactive training result on a random example to form an event-driven pre-reactive scheduling agent. According to the method provided by the invention, the optimal balance between the solving precision and the solving speed is realized, and the complex multi-target scheduling problem with various uncertain events can be solved by the proposed method.

Description

Technical Field

[0001] The present application relates to the intersection of deep reinforcement learning and dynamic workshop scheduling, and specifically to a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning. Background Art

[0002] As the integration between manufacturing and advanced technologies such as artificial intelligence becomes closer and closer, intelligent manufacturing has become one of the keys to the transformation and upgrading of traditional manufacturing and the transformation of manufacturing enterprises. As a key link in the manufacturing process, the production workshop urgently needs intelligent production scheduling methods. Such methods require the ability to handle random interference in real time, consider different goals at the same time, and have strong versatility and portability. Most scheduling problems existing in modern multi-variety and small-batch discrete manufacturing systems can be regarded as dynamic multi-objective flexible jobshop scheduling problems (Dynamic Multi-objective Flexible Jobshop Scheduling Problem, DMFJSP) problems, and are more difficult than traditional jobshop scheduling problems. The DMFJSP problem is highly complex and versatile, and is of great significance to research in academia and industry. Based on this, this application chooses DMFJSP as a typical dynamic scheduling method study, and conducts research on multi-objective intelligent scheduling of dynamic flexible workshops based on deep reinforcement learning.

[0003] In order to solve dynamic events, traditional DMFJSP scheduling methods are divided into two types: metaheuristic scheduling and rule scheduling. Metaheuristics can obtain nearly optimal solutions, but the time efficiency is low. When the system structure changes, it even needs to be redesigned, which is contrary to the concept of intelligent production. Rule scheduling can make accurate real-time reflections of dynamic interference, but most rules are short-sighted and the scheduling effect is worrying. In recent years, more work has begun to use deep reinforcement learning to solve dynamic problems. The basic idea of ​​this type of method is to use deep reinforcement learning agents to adaptively select feasible actions at different rescheduling points, thereby minimizing the impact of uncertain interference and optimizing long-term goals.

[0004] However, most of the work based on deep reinforcement learning is carried out for a single goal, and there is still little research on DMFJSP. In particular, for dynamic scheduling problems, most of them are based on fully reactive scheduling, and there is almost no research on event-driven pre-reactive scheduling, which has better effects and consumes less resources. For event-driven pre-reactive scheduling, its system process is not exactly the same as the basic Markov decision process of reinforcement learning, which is why there is a gap in this type of scheduling method based on deep reinforcement learning. Applying deep reinforcement learning to event-driven pre-reactive scheduling can not only realize intelligent scheduling, but also prevent the algorithm from easily falling into local optimality and wasting computing resources. Therefore, reconstructing the Markov decision process and applying deep reinforcement learning to event-driven-pre-reactive multi-objective dynamic flexible job shop problems is a technical problem that needs to be solved urgently. Summary of the invention

[0005] The embodiment of the present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, which achieves the best trade-off between solution accuracy and solution speed. The proposed method can handle complex multi-objective scheduling problems with multiple uncertain events.

[0006] To solve the above technical problems, in the first aspect, an embodiment of the present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, the method comprising the following steps: first, converting the deterministic scheduling problem into a Markov decision model; the conversion process comprises: establishing the optimization objective, the state space, the action space and the reward function of the Markov decision process; then, based on the Markov decision model, constructing a deterministic scheduling algorithm based on a multi-objective hybrid deep Q network and designing a training process to train the algorithm; testing the training results on random examples to form a deterministic scheduling model agent; next, constructing a Markov jump decision process, and converting the event-driven pre-reactive scheduling problem into a Markov jump decision process; the conversion process comprises: defining the Markov jump decision process and establishing the state and observation space, the action space and the reward function of the Markov jump decision process; finally, based on the hybrid deep recursive Q network algorithm, constructing a training process, and interactively training the event-driven pre-reactive scheduling agent with the simulation environment; testing the interactive training results on random examples to form an event-driven pre-reactive scheduling agent.

[0007] In some exemplary embodiments, the calculation formula of the optimization target is as follows:

[0008] Minimize makespan=max (makespan i ) (1)

[0009]

[0010] Among them, makespan is the makespan in all processes i The maximum value of J i For the i-th workpiece; makespan i For J i The completion time of the last process; Tard time Indicates the total delay time; D i Indicates J i Delay time; Tard num Indicates the total number of delayed workpieces; A variable between 0 and 1, used to describe whether to make span i >D i ; U ave represents the average machine utilization rate; M represents the machine; m is the total number of machines; n is the total number of workpieces; M k represents the kth machine; U k Indicates M k Twork represents the total load of all machines; Twork k Indicates M k of load.

[0011] In some exemplary embodiments, the parameters of the state space of the Markov decision process include: average machine utilization U ave , standard deviation of machine utilization U std , Average workpiece process scheduling ratio CRO ave , average job scheduling rate CRJ ave , standard deviation of average job scheduling rate CRJ std , estimated workpiece delay rate Tard num_e , the current actual workpiece delay rate Tard num_a and the maximum machine utilization U max ; The estimated makespan, current makespan, and the difference between the estimated makespan and the latest delivery date in the parameters are represented by C max_e , C max_a , C max_loss Indicates; the estimated delay time and the current actual delay time are respectively represented by Tard time_e , Tard time_a The estimated total machine load, the current actual total machine load and the number of delayed workpieces are represented by Twork e 、Twork a , Tard job express.

[0012] In some exemplary embodiments, the action space of the Markov decision process includes machine selection rules and workpiece selection rules; the establishment of the action space includes: combining the machine selection rules and the workpiece selection rules to obtain a 45-dimensional action space; the machine selection rule includes five cases, which are respectively recorded as: Mrule1, Mrule2, Mrule3, Mrule4, and Mrule5; Mrule1: select the machine that ends the earliest; Mrule2: select the machine with the shortest processing time; Mrule3: select the machine with the least load; Mrule4: select the machine with the lowest utilization rate; Mrule5: select the machine that becomes idle the earliest; the workpiece selection rule includes nine cases, which are respectively recorded as: Jrule 1, Jrule 2, Jrule 3, Jrule 4, Jrule 5, Jrule 6, Jrule 7, Jrule 8, and Jrule 9; Jrule 1: when there is no workpiece delay, select workpiece i that has scheduled j processes and satisfies Minimum; responsible for selecting the workpiece with the longest delay; Jrule 2: When there is no workpiece delay, select workpiece i that satisfies Minimum; otherwise, select the workpiece with the longest delay; Jrule 3: When there is no workpiece delay, select workpiece i that satisfies Minimum; Jrule 4: When there is no workpiece delay, select workpiece i that has scheduled j processes and satisfies Smallest; Jrule 5: Select the workpiece with the shortest processing time in the next process; Jrule 6: Select the workpiece with the longest remaining processing time; Jrule 7: Select the workpiece with the earliest delivery date; Jrule 8: When there is no workpiece delay, select the workpiece with the least remaining processing steps; otherwise, select the workpiece with the least remaining processing steps among the delayed workpieces; Jrule 9: Select the workpiece with the shortest remaining processing time.

[0013] In some exemplary embodiments, a deterministic scheduling algorithm based on a multi-objective hybrid deep Q network includes a duel network and a weight network; wherein the duel network includes nine fully connected layers, wherein the nine fully connected layers include an input layer, six hidden layers and two output layers; the weight network is used to dynamically allocate weights, including a gated recurrent unit and a Softmax activated output layer.

[0014] In some exemplary embodiments, the Markov jump decision process is defined as an octet including a state space, a jump transition function, a dynamic event, an observation space, an action space, a state transition function, a reward function, and a reward decay coefficient, denoted as: <S,J,E,Z,A,P,r,γ<。

[0015] In some exemplary embodiments, the characteristics of the state and observation space of the Markov jump decision process include 14 dimensions, namely: the average utilization rate of the machine in the scheduling plan, denoted as U ave ; The number of delayed jobs in the scheduling plan, denoted as Tard num ; The delay time in the scheduling plan is denoted as Tard time ; Scheduling plan Recorded as Makespan; the total machine load in the scheduling plan, recorded as Twork; the average processing time of all processes, recorded as OPT ave ; The time when the dynamic event occurs is recorded as DT; The average arrival time of the workpiece is recorded as Arrive ave ; The average failure time of the machine is recorded as Break ave ; The average idle time of the machine, recorded as Idle ave ; The average start time of the process, recorded as OPS ave ; The average end time of the process, denoted as OPE ave ; The type of dynamic event, if there is no dynamic event, it is 0, recorded as DET; The number of disturbed processes, recorded as DON.

[0016] In some exemplary embodiments, the action space of the Markov jump decision process includes 7 rescheduling rules; before performing rescheduling, partial rescheduling is subdivided according to whether the interrupted operation can be directly inserted into the idle time of the current state of the machine.

[0017] In some exemplary embodiments, the reward function of the Markov jump decision process is determined by the state feature and the observation feature; the expression of the reward function of the Markov jump decision process is as follows:

[0018] R(z t ,s t , e t , a t )=αR(s t , (z t+1 , e t+1 )|a t )+βR(z t ,(s t , a t )|e t ) (6)

[0019] Among them, R(s t , (z t+1 , e t+1 )|a t ) is the state reward function, R(z t ,(s t , a t )|et ) is the observation reward function;

[0020] In order to reduce the algorithm running time, the single action running time ART(t) is included in the reward function, and the result is:

[0021] R(z t ,s t , e t , a t )=αR(s t , (z t+1 , e t+1 )|a t )+βR(z t ,(s t , a t )|e t )-ART(t) (7)

[0022] Among them, ART(t) is the running time of a single action.

[0023] On the second aspect, the embodiment of the present application also provides a dynamic flexible job shop multi-objective scheduling model based on deep reinforcement learning, which is scheduled using the dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning as described in the above embodiment, including: a Markov decision model module, an algorithm construction and training module, a Markov jump decision process conversion module and a scheduling agent training module connected in sequence; the Markov decision model module is used to convert the deterministic scheduling problem into a Markov decision model; the conversion process includes: establishing optimization objectives, the state space, action space and reward function of the Markov decision process; the algorithm construction and training module is used to construct a deterministic scheduling based on a multi-objective hybrid deep Q network according to the Markov decision model. The algorithm is constructed and the training process is designed to train the algorithm; the training results are tested on random examples to form a deterministic scheduling model agent; the Markov jump decision process conversion module is used to construct the Markov jump decision process and convert the event-driven pre-reactive scheduling problem into the Markov jump decision process; the conversion process includes: the definition of the Markov jump decision process and the establishment of the state and observation space, action space and reward function of the Markov jump decision process; the scheduling agent training module is used to construct the training process according to the hybrid deep recursive Q network algorithm, and interactively train the event-driven pre-reactive scheduling agent with the simulation environment; the interactive training results are tested on random examples to form an event-driven pre-reactive scheduling agent.

[0024] The technical solution provided by the embodiments of the present application has at least the following advantages:

[0025] The embodiment of the present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, which includes the following steps: first, converting the deterministic scheduling problem into a Markov decision model; the conversion process includes: establishing an optimization objective, a state space, an action space and a reward function of the Markov decision process; then, based on the Markov decision model, constructing a deterministic scheduling algorithm based on a multi-objective hybrid deep Q network (MOHDQN) and designing a training process to train the algorithm; testing the training results on random examples to form a deterministic scheduling model agent; next, constructing a Markov jump decision process, and converting the event-driven pre-reactive scheduling problem into a Markov jump decision process; the conversion process includes: defining the Markov jump decision process and establishing the state and observation space, action space and reward function of the Markov jump decision process; finally, based on the hybrid deep recursive Q network (HDRQN) algorithm, constructing a training process, and interactively training an event-driven pre-reactive scheduling agent with a simulation environment; testing the interactive training results on random examples to form an event-driven pre-reactive scheduling agent.

[0026] The present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, which realizes event-driven pre-reaction dynamic scheduling. The present application constructs a deterministic scheduling method based on MOHDQN by converting the deterministic scheduling problem into a Markov decision model. Secondly, the constructed deterministic scheduling agent is trained as a scheduling scheme generation method and a rescheduling base method. Finally, a Markov jump decision process is constructed, and the event-driven pre-reaction scheduling process is converted into a Markov jump decision process, which is trained through the HDRQN algorithm to obtain an agent. The method of the present application achieves the best trade-off between solution accuracy and solution speed, and the proposed method can handle complex multi-objective scheduling problems with multiple uncertain events. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] One or more embodiments are exemplarily described by pictures in the corresponding drawings, and these exemplifications do not constitute limitations on the embodiments. Unless otherwise stated, the pictures in the drawings do not constitute proportional limitations.

[0028] Figure 1 A flow chart of a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning provided in an embodiment of the present application.

[0029] Figure 2 Schematic diagram of the overall architecture of the dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning provided in an embodiment of the present application.

[0030] Figure 3This is a diagram of the MOHDQN algorithm structure provided in an embodiment of the present application.

[0031] Figure 4 A schematic diagram of the Markov jump decision process provided in an embodiment of the present application.

[0032] Figure 5 This is a structural diagram of the HDRQN algorithm provided in an embodiment of the present application.

[0033] Figure 6 A schematic diagram of the structure of a dynamic flexible job shop multi-objective scheduling model based on deep reinforcement learning provided in an embodiment of the present application. DETAILED DESCRIPTION

[0034] As can be seen from the background technology, most of the existing work based on deep reinforcement learning is carried out for a single goal, and there is still little research on DMFJSP. In particular, for dynamic scheduling problems, most of them are based on fully reactive scheduling, and there is almost no research on event-driven pre-reactive scheduling with better effects and less resource consumption.

[0035] In order to solve the above technical problems, an embodiment of the present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, which includes the following steps: first, converting the deterministic scheduling problem into a Markov decision model; the conversion process includes: establishing an optimization objective, a state space, an action space and a reward function of the Markov decision process; then, based on the Markov decision model, constructing a deterministic scheduling algorithm based on a multi-objective hybrid deep Q network and designing a training process to train the algorithm; testing the training results on random examples to form a deterministic scheduling model agent; next, constructing a Markov jump decision process, and converting the event-driven pre-reactive scheduling problem into a Markov jump decision process; the conversion process includes: defining the Markov jump decision process and establishing the state and observation space, action space and reward function of the Markov jump decision process; finally, based on the hybrid deep recursive Q network algorithm, constructing a training process, and interactively training an event-driven pre-reactive scheduling agent with a simulation environment; testing the interactive training results on random examples to form an event-driven pre-reactive scheduling agent. The dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning provided in this application achieves the best trade-off between solution accuracy and solution speed. The proposed method can handle complex multi-objective scheduling problems with multiple uncertain events.

[0036] The following will describe the various embodiments of the present application in detail with reference to the accompanying drawings. However, it will be appreciated by those skilled in the art that in the various embodiments of the present application, many technical details are provided in order to enable the reader to better understand the present application. However, even without these technical details and various changes and modifications based on the following embodiments, the technical solution claimed in the present application can be implemented.

[0037] See also Figure 1 The embodiment of the present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, the method comprising the following steps:

[0038] Step S101, converting the deterministic scheduling problem into a Markov decision model; the conversion process includes: establishing an optimization target, a state space, an action space and a reward function of the Markov decision process.

[0039] Step S102: Based on the Markov decision model, a deterministic scheduling algorithm based on a multi-objective hybrid deep Q network (MOHDQN) is constructed and a training process is designed to train the algorithm; the training results are tested on random examples to form a deterministic scheduling model agent A. 1 .

[0040] Step S103, constructing a Markov jump decision process, and converting the event-driven pre-reactive scheduling problem into a Markov jump decision process; the conversion process includes: defining the Markov jump decision process and establishing the state and observation space, action space and reward function of the Markov jump decision process.

[0041] Step S104: Based on the hybrid deep recursive Q network (HDRQN) algorithm, a training process is constructed, and an event-driven pre-reactive scheduling agent is interactively trained with the simulation environment; the interactive training results are tested on random examples to form an event-driven pre-reactive scheduling agent A. 2 .

[0042] This application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, such as Figure 2 As shown in Figure 1, the deterministic scheduling problem is first converted into a Markov decision process, and a deterministic scheduling method based on MOHDQN (referred to as MOHDQN algorithm) is constructed to form a deterministic scheduling agent A. 1 Then, the constructed deterministic scheduling agent A 1 Then, the Markov jump decision process was constructed, and the event-driven pre-reactive scheduling process was converted into a Markov jump decision process. Finally, the event-driven pre-reactive scheduling agent A was obtained through training with the HDRQN algorithm. 2 .

[0043] Furthermore, the variables used in all the steps and their explanations are shown in Table 1 below.

[0044] Table 1 Variables and their related explanations

[0045]

[0046] Furthermore, the calculation formula of the optimization target in step S101 is as follows:

[0047] Minimize makespan=max (makespan i ) (1)

[0048]

[0049] Among them, makespan is the makespan in all processes i The maximum value of J i For the i-th workpiece; makespan i For J i The completion time of the last process; Tard time Indicates the total delay time; D i Indicates J i Delay time; Tard num Indicates the total number of delayed workpieces; A variable between 0 and 1, used to describe whether to make span i >D i ; U ave represents the average machine utilization rate; M represents the machine; m is the total number of machines; n is the total number of workpieces; M k represents the kth machine; U k Indicates M k Twork represents the total load of all machines; Twork k Indicates M k of load.

[0050] Furthermore, in step S101, the state space of the Markov decision process is shown in Table 2 below.

[0051] Table 2 State space definition

[0052]

[0053]

[0054] Furthermore, in step S101, the action space of the Markov decision process is as shown in Table 3 below, and a 45-dimensional action space is obtained by combining the machine selection rule and the workpiece selection rule.

[0055] Table 3 Action space definition

[0056]

[0057] Furthermore, the reward function R(s) in step S101 t ,a t ) is defined as the process shown in Algorithm 1.

[0058]

[0059]

[0060] In some embodiments, the deterministic scheduling algorithm based on the multi-objective hybrid deep Q network (MOHDQN) in step S102 includes a dueling q-network (denoted as Q) and a weight network Among them, the Q network includes nine fully connected layers, among which the nine fully connected layers include one input layer, six hidden layers and two output layers.

[0061] In order to alleviate the problem of overestimation of Q value, the Q network is copied to obtain the target network The weight network is used to dynamically allocate weights, including the gated recurrent unit (GRU) and the Softmax activation output layer. The specific structure is as follows Figure 3 As shown. For o optimization objectives, the weight network is mainly used to dynamically allocate weights W = (w 1 ,…,w o ).

[0062] Furthermore, the hyperparameters in step S102 are set to the values ​​shown in Table 4, and the training method is defined as the process shown in Algorithm 2.

[0063] Table 1. Definition of MOHDQN hyperparameters

[0064]

[0065]

[0066]

[0067] Furthermore, the Markov jump decision process proposed in step S103 is defined as an octet<S,J,E,Z,A,P,r,γ> :State space, jump transfer function, dynamic events, observation space, action space, state transfer function, reward function and reward attenuation coefficient. The specific process is as follows Figure 4 As shown. Since this application uses model-free reinforcement learning, the above transfer function is learned by the algorithm through environmental interaction. When the agent starts running, it first observes the initial observation Z 0 , if and only if a random event is triggered When the observation is distributed according to From Z 0 Transition to S 0 At this time, take action a 0 , state S 0 According to P(Z 1 |a 0 ,s 0 ) transition to the observed value Z 1 .

[0068] Furthermore, the characteristics of the state and observation space in step S103 include 14 dimensions, as shown in Table 5.

[0069] Table 2 Definition of state and observation space characteristics

[0070]

[0071] Furthermore, the action space in step S103 consists of 7 rescheduling rules. Before executing the rescheduling, the partial rescheduling is further subdivided according to whether the interrupted operation can be directly inserted into the idle time of the current state of the machine. Specifically, it is necessary to determine whether to reschedule only the interrupted operation of the job or to reschedule all subsequent operations. In order to record the workpiece and machine information, the machine information MIL and the workpiece information list JIL are constructed. These lists record detailed information such as the start and end time of the operation, the processing time, the assigned workpiece / machine, and the operations scheduled on each machine. When a dynamic event occurs at step t, the environment returns the disturbed operation set DO(t). Machine M k The idle time is defined as the list ILM k (t). For the k If all the processing steps are carried out on Then Put in ILM k (t). The specific judgment method is shown in Algorithm 3.

[0072]

[0073] The 7 rescheduling rules are as follows (1)-(7):

[0074] (1) Partial rescheduling rule 1. When there is any insertable rescheduling process, the insertable process is inserted into the earliest machine that can be inserted, and other processes execute the right shift rescheduling rule. The specific process of partial rescheduling rule 1 is shown in Algorithm 4.

[0075]

[0076] (2) Partial rescheduling rule 2. When there is any insertable rescheduling process, the insertable process is inserted into the machine with the shortest operation time, and the other processes execute the right shift rescheduling rule. The specific process of partial rescheduling rule 2 is shown in Algorithm 5.

[0077]

[0078] (3) Hybrid Partial Rescheduling Rule 1. The insertable scheduling process is executed in Algorithm 4, and the other processes are partially rescheduled using the MOHDQN algorithm in S2. The specific hybrid partial rescheduling rule 1 is shown in Algorithm 6.

[0079]

[0080] (4) Hybrid Partial Rescheduling Rule 2. The insertable scheduling process is executed in Algorithm 5, and the other processes are partially rescheduled using the MOHDQN algorithm in S2. The specific hybrid partial rescheduling rule 2 is shown in Algorithm 7.

[0081]

[0082]

[0083] (5) Complete rescheduling rules. The specific complete rescheduling rules are shown in the process of Algorithm 8.

[0084]

[0085] (6)Right shift rescheduling rules.

[0086] (7) Partial rescheduling rule 3. Rescheduling is performed on all affected processes and their subsequent processes. The specific process of partial rescheduling rule 3 is shown in Algorithm 9.

[0087]

[0088] Furthermore, the reward function in step S103 needs to be determined by combining the state characteristics and the observation characteristics. The state reward function is defined as R(s t ,(zt+1 ,e t+1 )|a t ), the observation reward function is defined as R(z t ,(s t ,a t )|e t ), the two reward functions follow the same calculation method as shown in Algorithm 10. The comprehensive reward is R(z t ,s t ,e t ,a t )=αR(s t ,(z t+1 ,e t+1 )|a t )+βR(z t ,(s t ,a t )|e t ). In addition, in order to reduce the algorithm running time as much as possible, the single action running time ART(t) is included in the reward function, then R(z t ,s t ,e t ,a t )=αR(s t ,(z t+1 ,e t+1 )|a t )+βR(z t ,(s t ,a t )|e t )-ART(t).

[0089]

[0090]

[0091] Furthermore, the HDRQN algorithm in step S104 includes a GRU as the input layer with a dimension of 28, which is the concatenation of the state and observation space. The hidden layer is a 3-layer fully connected layer with a dimension of 64. In the output layer, the output layer dimension of the advantage network is 7, and the output layer dimension of the value network is 1, which is finally integrated into a 7-dimensional Q value vector. The hyperparameter definition is shown in Table 6. The HDRQN algorithm structure diagram is shown in Figure 5 As shown, the training method is defined as the process shown in Algorithm 11.

[0092] Table 3 HDRQN hyperparameter definitions

[0093]

[0094]

[0095] The data set in this embodiment is randomly generated by a function, and the characteristics of this data set are shown in Table 7. In addition, the dynamic events in this embodiment are defined as external events that cause changes to the original plan, specifically as shown in (1)-(3) below:

[0096] (1) Workpiece late. If and only if S i0 ≤t, workpiece J i has not arrived and is considered to be triggered by a dynamic event.

[0097] (2) Workpiece early. If and only if Ed k (OM i0k -1)<t, workpiece J i has arrived and is considered to be triggered by a dynamic event. Where Ed k (x) represents the end time of the xth operation on machine k.

[0098] (3) When the load on the machine exceeds I and there are multiple subsequent processes waiting to be processed on the machine, it is considered to be triggered by a dynamic event. During maintenance, if there is exactly a process in production, it will continue to produce after inspection.

[0099] Table 4 Characteristics of Random Data Set Generation

[0100]

[0101] See Figure 6The embodiment of the present application also provides a dynamic flexible job shop multi-objective scheduling model based on deep reinforcement learning, which is scheduled using the dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning as described in the above embodiment, including: a Markov decision model module 101, an algorithm construction and training module 102, a Markov jump decision process conversion module 103 and a scheduling agent training module 104 connected in sequence; the Markov decision model module 101 is used to convert the deterministic scheduling problem into a Markov decision model; the conversion process includes: establishing an optimization objective, a state space, an action space and a reward function of the Markov decision process; the algorithm construction and training module 102 is used to construct a deterministic scheduling problem based on MOHDQN according to the Markov decision model. A qualitative scheduling algorithm is designed and a training process is designed to train the algorithm; the training results are tested on random examples to form a deterministic scheduling model agent; the Markov jump decision process conversion module 103 is used to construct a Markov jump decision process and convert the event-driven pre-reactive scheduling problem into a Markov jump decision process; the conversion process includes: the definition of the Markov jump decision process and the establishment of the state and observation space, action space and reward function of the Markov jump decision process; the scheduling agent training module 104 is used to construct a training process according to the HDRQN algorithm, and interactively train the event-driven pre-reactive scheduling agent with the simulation environment; the interactive training results are tested on random examples to form an event-driven pre-reactive scheduling agent.

[0102] In order to prove the scientificity and accuracy of the method and model, this application first compares the deterministic scheduling method based on the multi-objective hybrid deep Q network (MOHDQN) with the well-known scheduling rules, composite scheduling rules, random sample results, and the widely recognized multi-objective algorithm NSGA-II. NSGA-II was tested in 10, 25, and 50 iterations, respectively, and the population size was set to 200. The result of random sampling is the average of 500 random samples. The evaluation indicators are the inverse generation distance (IGD), hypervolume (HV), and the running time required for each method to generate a schedule. The comparison results are shown in Tables 8 to 12.

[0103] Table 5 Comparison between MOHDQN and simple rule IGD

[0104]

[0105] Table 6 Comparison of MOHDQN with NSGA-II and combination rule IGD

[0106]

[0107]

[0108] IGD evaluates how close the solutions generated by an optimization algorithm are to the theoretical optimal frontier. The lower the IGD, the better the algorithm performance is. The algorithms with the lowest IGD values ​​are highlighted in bold in Tables 8 and 9. The proposed method consistently outperforms rule-based and composite rule-based methods in all sample sets. Compared with NSGA-II with only 10 iterations, MOHDQN achieves better performance in most cases and remains competitive compared with NSGA-II with 25 iterations. In addition, when dealing with large instances, such as those involving 50 or more machines and jobs, both MOHDQN and NSGA-II with 50 iterations are able to achieve the optimization goal. Although random sampling has lower variance, it also has lower accuracy. MOHDQN exhibits strong stability and consistently provides acceptable Pareto solutions in a variety of samples.

[0109] Table 7 Comparison between MOHDQN and simple rule HV

[0110]

[0111]

[0112] Table 8 Comparison of MOHDQN with NSGA-II and combination rule HV

[0113]

[0114] HV measures the volume between the result and the reference point (usually the worst point in the objective space). The larger the HV value, the larger the area (or volume) occupied by the solution in the objective space, which generally means that the diversity and quality of the solution are better. Under the HV index, the performance of MOHDQN is more obvious than that under the IGD index, and this advantage is mainly reflected in large-scale examples. It is worth noting that the HV value of NSGA-II with 50 iterations is very close to MOHDQN. If a larger population size and more iterations are used, it is reasonable to consider that NSGA-II may match or even exceed MOHDQN. However, algorithms that require a lot of running time are not suitable for rapidly changing production environments, nor are they suitable as the basis for event-driven predictive response scheduling.

[0115] Table 9 Comparison of running time of MOHDQ algorithm, rules and NSGA-II (seconds)

[0116]

[0117] From the above analysis, it can be concluded that the method of this application is generally better than the rule-based method and NSGA-II with 10 iterations and 25 iterations in most scenarios. However, in some scenarios and some indicators, it is slightly inferior to NSGA-II with 50 iterations. In addition, the MOHDQN method in this application is mainly used as a generator of scheduling schemes and a basic algorithm for rescheduling, and its running time is as important as Pareto optimality. Therefore, this application compares the running time of different algorithms, and the results are shown in Table 11. For problems smaller than 50×40, MOHDQN is the fastest method; beyond this scale, it is faster than NSGA-II with 10 iterations. However, for larger problem sizes, the accuracy of NSGA-II after only 10 iterations can no longer be guaranteed. Although more iterations improve accuracy, they significantly increase the computation time. NSGA-II runs several or even dozens of times slower than MOHDQN at 50 iterations. Although the rule-based method is also fast, its performance varies greatly between different instances. Therefore, the method proposed in this application achieves the best trade-off between accuracy and running speed, and is more suitable as a base algorithm for subsequent event-driven pre-reaction scheduling.

[0118] After verifying the effectiveness of the deterministic scheduling method, this embodiment will continue to evaluate the performance of the event-driven pre-reactive scheduling method HDRQN from three aspects: IGD, HV, and running time. The calculation results are shown in Tables 13 to 15.

[0119] Table 10 Comparison between HDRQN and regular IGD

[0120]

[0121] In Table 13, the full rescheduling rule, as a method that considers the problem from a global perspective, shows a strong advantage regardless of the examples. Although HDRQN performs as well as or better than the full rescheduling rule on some examples, the overall performance of the full rescheduling rule is still better. Outside of the full rescheduling rule, the proposed HDRQN shows the best and most stable performance, outperforming other methods in the vast majority of examples. The right-shift rescheduling rule also performs well, although its performance and stability lag behind HDRQN, and this gap widens further when tested on a larger set of examples.

[0122] Table 11 Comparison between HDRQN and regular HV

[0123]

[0124] In Table 14, the full rescheduling rule still shows excellent performance advantages regardless of the scale of the example, but the performance of the full rescheduling rule is at the expense of operating efficiency, as shown in Table 16. In addition to the full rescheduling rule, HDRQN has the best accuracy, and even exceeds the full rescheduling rule in some examples. Although the partial rescheduling rule 1 has the best stability, its accuracy is poor. The right shift rescheduling rule HV also performs well and can play a certain role in some specific scenarios or industries.

[0125] Table 12 Comparison of HDRQN and rule running time

[0126]

[0127]

[0128] As shown in Table 15, among all the methods, the calculation time of the full rescheduling rule is the longest, the calculation speed of the right-shift rescheduling rule is the fastest, and HDRQN is between the two. As the sample size increases, the calculation time of the full rescheduling rule increases exponentially, which may become more serious in a complex actual production environment. Therefore, based on Tables 13 to 15, the HDRQN method proposed in this application achieves the best balance between running time and accuracy. However, this does not reduce the value of other rules. Each method has a specific production environment and a use case that is more suitable for it. HDRQN, the full rescheduling rule, and the right-shift rescheduling rule each have their own advantages and disadvantages. HDRQN provides the best balance between running time and accuracy, the full rescheduling rule achieves the highest accuracy, and the right-shift rescheduling rule is the fastest. For most medium-sized and large enterprises, such as automobile manufacturing, printing industry, etc., dynamic events are frequent and require more efficient scheduling solutions without significantly reducing accuracy. These industries are usually faced with high production complexity and resource constraints, so HDRQN is the most suitable choice. For companies with fewer dynamic events and lower production complexity, such as aerospace or medical device manufacturers, the full rescheduling rule is the best choice due to its superior accuracy. On the other hand, smaller companies with limited computing power and expertise may benefit more from the right-shift rescheduling rule because it provides simpler and faster adjustments to the original production plan.

[0129] Based on the above technical scheme, the embodiment of the present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, which includes the following steps: first, the deterministic scheduling problem is converted into a Markov decision model; the conversion process includes: establishing the optimization objective, the state space, action space and reward function of the Markov decision process; then, based on the Markov decision model, a deterministic scheduling algorithm based on MOHDQN is constructed and a training process is designed to train the algorithm; the training results are tested on random examples to form a deterministic scheduling model agent; next, a Markov jump decision process is constructed, and the event-driven pre-reactive scheduling problem is converted into a Markov jump decision process; the conversion process includes: the definition of the Markov jump decision process and the establishment of the state and observation space, action space and reward function of the Markov jump decision process; finally, based on the HDRQN algorithm, a training process is constructed, and the event-driven pre-reactive scheduling agent is interactively trained with the simulation environment; the interactive training results are tested on random examples to form an event-driven pre-reactive scheduling agent.

[0130] The present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, which realizes event-driven pre-reaction dynamic scheduling. The present application constructs a deterministic scheduling method based on MOHDQN by converting the deterministic scheduling problem into a Markov decision model. Secondly, the constructed deterministic scheduling agent is trained as a scheduling scheme generation method and a rescheduling base method. Finally, a Markov jump decision process is constructed, and the event-driven pre-reaction scheduling process is converted into a Markov jump decision process, which is trained through the HDRQN algorithm to obtain an agent. The method of the present application achieves the best trade-off between solution accuracy and solution speed, and the proposed method can handle complex multi-objective scheduling problems with multiple uncertain events.

[0131] Those skilled in the art can understand that the above-mentioned embodiments are specific examples for implementing the present application, and in practical applications, various changes can be made to them in form and detail without departing from the spirit and scope of the present application. Any person skilled in the art can make their own changes and modifications without departing from the spirit and scope of the present application, so the scope of protection of the present application shall be based on the scope defined in the claims.

Claims

1. A dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, characterized in that: The following steps are involved: Transform the deterministic scheduling problem into a Markov decision model; The conversion process includes: establishing the optimization objective, the state space, action space and reward function of the Markov decision process; Based on the Markov decision model, a deterministic scheduling algorithm based on a multi-objective hybrid deep Q network is constructed and a training process is designed to train the algorithm; the training results are tested on random examples to form a deterministic scheduling model agent; Construct a Markov jump decision process and convert the event-driven pre-reactive scheduling problem into a Markov jump decision process; the conversion process includes: defining the Markov jump decision process and establishing the state and observation space, action space and reward function of the Markov jump decision process; Based on the hybrid deep recursive Q-network algorithm, a training process is constructed, and an event-driven pre-reactive scheduling agent is interactively trained with the simulation environment. The interactive training results are tested on random examples to form an event-driven pre-reactive scheduling agent.

2. The dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The calculation formula of the optimization target is as follows: Minimize makespan=max (makespan i ) (1) Among them, makespan is the makespan in all processes i The maximum value of J i For the i-th workpiece; makespan i For J i The completion time of the last process; Tard time Indicates the total delay time; D i Indicates J i Delay time; Tard num Indicates the total number of delayed workpieces; A variable between 0 and 1, used to describe whether to make span i >D i ; U ave represents the average machine utilization rate; M represents the machine; m is the total number of machines; n is the total number of workpieces; M k represents the kth machine; U k Indicates M k Twork represents the total load of all machines; Twork k Indicates M k of load.

3. The dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The parameters of the state space of the Markov decision process include: Average machine utilization U ave , standard deviation of machine utilization U std , Average workpiece process scheduling ratio CRO ave , average job scheduling rate CRJ ave , standard deviation of average job scheduling rate CRJ std , estimated workpiece delay rate Tard num_e , the current actual workpiece delay rate Tard num_a and the maximum machine utilization U max ; The estimated makespan, current makespan, and the difference between the estimated makespan and the latest delivery date in the parameters are represented by C max_e , C max_a , C max_loss Indicates; the estimated delay time and the current actual delay time are respectively represented by Tard time_e , Tard time_a The estimated total machine load, the current actual total machine load and the number of delayed workpieces are represented by Twork e 、Twork a , Tard job express.

4. The dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The action space of the Markov decision process includes a machine selection rule and a workpiece selection rule; the establishment of the action space includes: combining the machine selection rule and the workpiece selection rule to obtain a 45-dimensional action space; The machine selection rules include five cases, which are respectively recorded as: Mrule1, Mrule2, Mrule3, Mrule4, and Mrule5; Mrule1: Select the machine that finishes the earliest; Mrule2: Select the machine with the shortest processing time; Mrule3: Select the machine with the least load; Mrule4: Select the machine with the lowest utilization; Mrule5: Select the first idle machine; The workpiece selection rule includes nine cases, which are respectively recorded as: Jrule 1, Jrule 2, Jrule 3, Jrule 4, Jrule 5, Jrule 6, Jrule 7, Jrule 8, and Jrule 9; Jrule 1: When there is no workpiece delay, select workpiece i that has scheduled j processes and satisfies Minimum; Responsible for selecting the artifact with the longest delay; Jrule 2: When there is no workpiece delay, select workpiece i to satisfy minimum; otherwise, the workpiece with the longest delay is selected; Jrule 3: When there is no workpiece delay, select workpiece i to satisfy Minimum; Jrule 4: When there is no workpiece delay, select workpiece i that has scheduled j processes and satisfies Minimum; Jrule 5: Select the workpiece with the shortest processing time in the next process; Jrule 6: Select the workpiece with the longest remaining processing time; Jrule7: Select the workpiece with the earliest delivery date; Jrule 8: When there is no workpiece delay, select the workpiece with the least remaining processing steps; otherwise, select the workpiece with the least remaining processing steps among the delayed workpieces; Jrule 9: Select the workpiece with the shortest remaining processing time.

5. The dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The deterministic scheduling algorithm based on the multi-objective hybrid deep Q network includes a duel network and a weight network; wherein, The duel network includes nine fully connected layers, wherein the nine fully connected layers include one input layer, six hidden layers and two output layers; The weight network is used to dynamically allocate weights and includes a gated recurrent unit and a Softmax activated output layer.

6. The dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The Markov jump decision process is defined as an octet including state space, jump transition function, dynamic events, observation space, action space, state transition function, reward function and reward decay coefficient, denoted as: <S,J,E,Z,A,P,r,γ> .

7. The dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The characteristics of the state and observation space of the Markov jump decision process include 14 dimensions, namely: The average utilization rate of the machines in the scheduling plan is denoted by U ave ; The number of delayed jobs in the scheduling plan, denoted as Tard num ; The delay time in the scheduling plan is denoted as Tard time ; In the scheduling plan Denoted as Makespan; The total machine load in the scheduling plan is denoted as Twork; The average processing time of all processes is denoted as OPT ave ; The time when the dynamic event occurs is denoted as DT; The average arrival time of the workpiece is denoted as Arrive ave ; The average failure time of the machine is recorded as Break ave ; The average idle time of the machine, recorded as Idle ave ; The average start time of the process, recorded as OPS ave ; The average end time of the process, denoted as OPE ave ; The type of dynamic event, if there is no dynamic event, it is 0, recorded as DET; The number of disturbed processes is denoted as DON.

8. The dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The action space of the Markov jump decision process includes 7 rescheduling rules; Before executing the rescheduling, the partial rescheduling is subdivided according to whether the interrupted operation can be directly inserted into the idle time of the current state of the machine.

9. The dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning according to claim 1 is characterized in that: The reward function of the Markov jump decision process is determined by state features and observation features; The expression of the reward function of the Markov jump decision process is as follows: R(z t ,s t ,e t ,a t )=αR(s t ,(z t+1 ,e t+1 )|a t )+βR(z t ,(s t ,a t )|e t ) (6) Among them, R(s t ,(z t+1 ,e t+1 )|a t ) is the state reward function, R(z t ,(s t ,a t )|e t ) is the observation reward function; In order to reduce the algorithm running time, the single action running time ART(t) is included in the reward function, and the result is: R(z t ,s t ,e t ,a t )=αR(s t ,(z t+1 ,e t+1 )|a t )+βR(z t ,(s t ,a t )|e t )-ART(t) (7) Among them, ART(t) is the running time of a single action.

10. A dynamic flexible job shop multi-objective scheduling model based on deep reinforcement learning, characterized in that: The method for multi-objective scheduling of a dynamic flexible job shop based on deep reinforcement learning as described in any one of claims 1 to 9 is used for scheduling, comprising: a Markov decision model module, an algorithm construction and training module, a Markov jump decision process conversion module and a scheduling agent training module connected in sequence; The Markov decision model module is used to transform the deterministic scheduling problem into a Markov decision model; the transformation process includes: establishing an optimization objective, a state space, an action space and a reward function of the Markov decision process; The algorithm construction and training module is used to construct a deterministic scheduling algorithm based on a multi-objective hybrid deep Q network according to the Markov decision model and design a training process to train the algorithm; the training results are tested on random examples to form a deterministic scheduling model agent; The Markov jump decision process conversion module is used to construct a Markov jump decision process and convert the event-driven pre-reactive scheduling problem into a Markov jump decision process; the conversion process includes: defining the Markov jump decision process and establishing the state and observation space, action space and reward function of the Markov jump decision process; The scheduling agent training module is used to construct a training process according to a hybrid deep recursive Q network algorithm, and interactively train an event-driven pre-reactive scheduling agent with a simulation environment; the interactive training results are tested on random examples to form an event-driven pre-reactive scheduling agent.

Images