Dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning

Abstract

The application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, comprising the following steps: converting a deterministic scheduling problem into a Markov decision model; based on the Markov decision model, constructing a MOHDQN-based deterministic scheduling algorithm and designing a training process, training the algorithm, testing on a random example, and forming a deterministic scheduling model agent; constructing a Markov jump decision process, and converting an event-driven pre-responding scheduling problem into a Markov jump decision process; based on the HDRQN algorithm, constructing a training process, and training an event-driven pre-responding scheduling agent in interaction with a simulation environment; testing the interaction training result on a random example, and forming an event-driven pre-responding scheduling agent. The method of the application realizes the best trade-off in solving precision and solving speed, and the proposed method can process complex multi-objective scheduling problems with multiple uncertain events.

Description

TECHNICAL FIELD

[0001] The application relates to the cross field of deep reinforcement learning and dynamic job shop scheduling, in particular to a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning. BACKGROUND

[0002] With the combination of manufacturing industry and advanced technologies such as artificial intelligence becoming closer and closer, intelligent manufacturing has become one of the keys to the transformation and upgrading of traditional manufacturing industry and the transformation of manufacturing enterprises. As a key link in the manufacturing process, production workshop urgently needs intelligent production scheduling methods. Such methods require real-time processing of random disturbances, considering different objectives, and having strong versatility and portability. Most of the scheduling problems existing in modern multi-variety and small-batch discrete manufacturing systems can be regarded as dynamic multi-objective flexible job shop scheduling problem (DMFJSP) problems, and are more difficult than traditional job shop scheduling problems. DMFJSP problems have high complexity and versatility, and are of great significance to the research of academia and industry. Based on this, the present application selects DMFJSP as a typical dynamic scheduling method, and studies dynamic flexible job shop multi-objective intelligent scheduling based on deep reinforcement learning.

[0003] In order to solve dynamic events, traditional DMFJSP scheduling methods are divided into two kinds, meta-heuristic scheduling and rule scheduling. Meta-heuristic can obtain a nearly optimal solution, but the time efficiency is low, and when the system structure changes, it even needs to be redesigned, which is contrary to the concept of intelligent production. Rule scheduling can accurately reflect the real-time response to dynamic disturbances, but most rules are short-sighted, and the effect of scheduling is questionable. In recent years, more and more work has begun to use deep reinforcement learning to solve dynamic problems. The basic idea of such methods is to use deep reinforcement learning agents to adaptively select feasible actions at different rescheduling points, so as to minimize the impact of uncertain disturbances and optimize long-term goals.

[0004] However, most of the deep reinforcement learning-based work is carried out for single objective, and the research on DMFJSP is still less. Especially for dynamic scheduling problem, most of them are based on full reactive scheduling, and the research on event-driven pre-reactive scheduling which is better in effect and less in resource consumption is almost blank. For event-driven pre-reactive scheduling, the system flow is not completely the same as the basic Markov decision process of reinforcement learning, which is the reason why there is a blank in the deep reinforcement learning-based method for this type of scheduling method. Applying deep reinforcement learning to event-driven pre-reactive scheduling can not only realize intelligent scheduling, but also prevent the algorithm from easily falling into local optimum and wasting computing resources. Therefore, it is a technical problem to be solved to reconstruct the Markov decision process and apply deep reinforcement learning to event-driven pre-reactive multi-objective dynamic flexible job shop problem. SUMMARY

[0005] The embodiment of the present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, which realizes the best trade-off in solving precision and solving speed, and the proposed method can process complex multi-objective scheduling problems with multiple uncertain events.

[0006] To solve the above technical problems, in a first aspect, the embodiment of the present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, which comprises the following steps: first, converting the deterministic scheduling problem into a Markov decision model; the conversion process includes: establishing the optimization objective, the state space, the action space and the reward function of the Markov decision process; then, based on the Markov decision model, constructing a deterministic scheduling algorithm based on multi-objective hybrid deep Q network and designing a training process to train the algorithm; testing the training result on a random example to form a deterministic scheduling model agent; next, constructing a Markov jump decision process, and converting the event-driven pre-reactive scheduling problem into a Markov jump decision process; the conversion process includes: definition of the Markov jump decision process and establishment of the state and observation space, the action space and the reward function of the Markov jump decision process; finally, based on the hybrid deep recursive Q network algorithm, constructing a training process, and training the event-driven pre-reactive scheduling agent with the simulation environment; testing the interactive training result on a random example to form an event-driven pre-reactive scheduling agent.

[0007] In some example embodiments, the calculation formula of the optimization objective is as follows:

[0008] Minimize makespan=max (makespan i ) (1)

[0009]

[0010] Where, makespan refers to the makespan in all processes. i The maximum value of J; i For the i-th workpiece; makespan i For J i The completion time of the final process; Tard time Indicates the total delay time; D i J represents i Delay time; Tard num This indicates the total number of overdue workpieces; A variable between 0 and 1, used to describe whether makespan is active. i >D i ;U ave M represents the average machine utilization rate; m represents the number of machines; n represents the total number of workpieces; M k U represents the k-th machine; k M represents k Twork represents the utilization rate of all machines; Twork represents the total load of all machines; Twork k M represents k The load.

[0011] In some exemplary embodiments, the parameters of the state space of the Markov decision process include: average machine utilization U ave Standard deviation of machine utilization U std Average workpiece process scheduling ratio (CRO) ave Average workpiece scheduling rate CRJ ave Standard deviation of average workpiece scheduling rate CRJ std Estimated workpiece delay rate Tard num_e Current actual workpiece delay rate Tard num_a And the maximum value of machine utilization U max The estimated makespan, the current makespan, and the difference between the estimated makespan and the latest delivery date are represented by C, respectively. max_e C max_a C max_loss The estimated delay time and the current actual delay time are represented by Tard, respectively. time_e Tard time_a The estimated total machine load, the current actual total machine load, and the number of delayed jobs are represented by Twork, respectively. e Twork a Tard job express.

[0012] In some example embodiments, the action space of the Markov decision process comprises machine selection rules and job selection rules; the establishment of the action space comprises: combining the machine selection rules and the job selection rules to obtain a 45-dimensional action space; the machine selection rules comprise five cases, respectively denoted as: Mrule1, Mrule2, Mrule3, Mrule4, and Mrule5; Mrule1: selecting the machine with the earliest end; Mrule2: selecting the machine with the shortest processing time; Mrule3: selecting the machine with the least load; Mrule4: selecting the machine with the lowest utilization rate; and Mrule5: selecting the machine with the earliest idle; the job selection rules comprise nine cases, respectively denoted as: Jrule 1, Jrule 2, Jrule 3, Jrule 4, Jrule 5, Jrule 6, Jrule 7, Jrule 8, and Jrule 9; Jrule 1: when there is no job delay, selecting the job i that has been scheduled for j processes and satisfies ; being responsible for selecting the job with the longest delay; Jrule 2: when there is no job delay, selecting the job i that satisfies ; otherwise, selecting the job with the longest delay; Jrule 3: when there is no job delay, selecting the job i that satisfies ; Jrule 4: when there is no job delay, selecting the job i that has been scheduled for j processes and satisfies ; Jrule 5: selecting the job with the shortest processing time for the next process; Jrule 6: selecting the job with the longest remaining processing time; Jrule 7: selecting the job with the earliest delivery date; Jrule 8: when there is no job delay, selecting the job with the least remaining processing processes; otherwise, selecting the job with the least remaining processing processes among the delayed jobs; and Jrule 9: selecting the job with the shortest remaining processing time.

[0013] In some example embodiments, the deterministic scheduling algorithm based on the multi-objective hybrid deep Q network comprises a dueling network and a weight network; wherein the dueling network comprises nine fully connected layers, wherein the nine fully connected layers comprise one input layer, six hidden layers, and two output layers; and the weight network is used for dynamically allocating weights, comprising a gated recurrent unit and a Softmax activation output layer.

[0014] In some example embodiments, the Markov jump decision process is defined as an eight-tuple comprising a state space, a jump transition function, a dynamic event, an observation space, an action space, a state transition function, a reward function, and a reward decay coefficient, denoted as: <S, J, E, Z, A, P, r, γ<.

[0015] In some example embodiments, the state of the Markov jump decision process and the features of the observation space include 14 dimensions, respectively: the average utilization of machines in the schedule, denoted as U ave ; the number of tardy jobs in the schedule, denoted as Tard num ; the tardiness time in the schedule, denoted as Tard time ; the makespan in the schedule, denoted as Makespan; the total load of machines in the schedule, denoted as Twork; the average processing time of all operations, denoted as OPT ave ; the time of dynamic events, denoted as DT; the average arrival time of jobs, denoted as Arrive ave ; the average breakdown time of machines, denoted as Break ave ; the average idle time of machines, denoted as Idle ave ; the average start time of operations, denoted as OPS ave ; the average end time of operations, denoted as OPE ave ; the type of dynamic events, 0 if there is no dynamic event, denoted as DET; the number of disturbed operations, denoted as DON.

[0016] In some example embodiments, the action space of the Markov jump decision process includes 7 rescheduling rules; before performing rescheduling, the partial rescheduling is subdivided according to whether the interrupted operation can be directly inserted into the idle time of the current state of the machine.

[0017] In some example embodiments, the reward function of the Markov jump decision process is determined by the state features and the observation features; the expression of the reward function of the Markov jump decision process is as follows:

[0018] R(z t , s t , e t , a t ) = aR(s t , (z t+1 , e t+1 ) | a t ) + bR(z t , (s t , a t ) | e t ) (6)

[0019] wherein R(s t , (z t+1 , e t+1 ) | a t ) is the state reward function, R(z t , (s t , a t ) | et ) is an observation reward function;

[0020] In order to reduce the time consumption of the algorithm, the single action running time ART(t) is introduced into the reward function, and then the following equation is obtained:

[0021] R(z t , s t , e t , a t ) = aR(s t , (z t+1 , e t+1 ) | a t ) + bR(z t , (s t , a t ) | e t ) - ART(t) (7)

[0022] wherein ART(t) is the single action running time.

[0023] In a second aspect, the embodiments of the present application also provide a dynamic flexible job shop multi-objective scheduling model based on deep reinforcement learning, which adopts the dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning as described in the above embodiments for scheduling, and includes a Markov decision model module, an algorithm construction and training module, a Markov jump decision process conversion module and a scheduling agent training module connected in sequence; the Markov decision model module is used for converting a deterministic scheduling problem into a Markov decision model; the conversion process includes establishing an optimization target, a state space, an action space and a reward function of the Markov decision process; the algorithm construction and training module is used for constructing a deterministic scheduling algorithm based on a multi-objective hybrid deep Q network and designing a training process according to the Markov decision model, and training the algorithm; the training result is tested on a random example to form a deterministic scheduling model agent; the Markov jump decision process conversion module is used for constructing a Markov jump decision process, and converting an event-driven pre-emptive scheduling problem into a Markov jump decision process; the conversion process includes defining the Markov jump decision process, and establishing a state and observation space, an action space and a reward function of the Markov jump decision process; the scheduling agent training module is used for constructing a training process according to a hybrid deep recursive Q network algorithm, and training an event-driven pre-emptive scheduling agent by interacting with a simulation environment; the interactive training result is tested on a random example to form an event-driven pre-emptive scheduling agent.

[0024] The technical solutions provided by the embodiments of the present application have at least the following advantages:

[0025] The embodiment of the present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, which comprises the following steps: firstly, converting a deterministic scheduling problem into a Markov decision model; the conversion process comprises: establishing an optimization target, a state space, an action space and a reward function of the Markov decision process; then, based on the Markov decision model, a deterministic scheduling algorithm based on a multi-objective hybrid deep Q network (MOHDQN) is constructed, and a training process is designed to train the algorithm; the training result is tested on a random example to form a deterministic scheduling model agent; next, a Markov jump decision process is constructed, and an event-driven pre-reactive scheduling problem is converted into the Markov jump decision process; the conversion process comprises: definition of the Markov jump decision process and establishment of the state and observation space, the action space and the reward function of the Markov jump decision process; finally, based on a hybrid deep recursive Q network (HDRQN) algorithm, a training process is constructed, and an event-driven pre-reactive scheduling agent is trained in interaction with a simulation environment; the interactive training result is tested on a random example to form an event-driven pre-reactive scheduling agent.

[0026] The present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, which realizes event-driven pre-reactive dynamic scheduling. The present application converts a deterministic scheduling problem into a Markov decision model, and constructs a deterministic scheduling method based on MOHDQN. Secondly, the constructed deterministic scheduling agent is trained as a scheduling scheme generation method and a rescheduling base method. Finally, a Markov jump decision process is constructed, and an event-driven pre-reactive scheduling process is converted into the Markov jump decision process, which is trained by an HDRQN algorithm to obtain an agent. The method of the present application realizes the best trade-off in solving precision and solving speed, and the proposed method can process complex multi-objective scheduling problems with multiple uncertain events. BRIEF DESCRIPTION OF DRAWINGS

[0027] One or more embodiments are illustrated by way of example in the accompanying drawings that are not intended to be limiting of the embodiments so far as is not otherwise specifically recited in the present application, the drawings of which form a part of this disclosure. In the drawings:

[0028] Figure 1 A flowchart of a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning provided by the embodiment of the present application is shown.

[0029] Figure 2 A schematic diagram of the overall architecture of a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning provided by the embodiment of the present application is shown.

[0030] Figure 3A structure diagram of the MOHDQN algorithm provided by the embodiment of the application.

[0031] Figure 4 A Markov jump decision process schematic diagram provided by the embodiment of the application.

[0032] Figure 5 A structure diagram of the HDRQN algorithm provided by the embodiment of the application.

[0033] Figure 6 A structure schematic diagram of a dynamic flexible job shop multi-objective scheduling model based on deep reinforcement learning provided by the embodiment of the application. DETAILED DESCRIPTION

[0034] As known from the background, most of the existing deep reinforcement learning-based jobs are developed for single objectives, and the research on DMFJSP is still less. In particular, for the dynamic scheduling problem, most of them are developed based on complete reactive scheduling, and the research on event-driven pre-reactive scheduling which is better in effect and less in resource consumption is almost blank.

[0035] In order to solve the above technical problems, the embodiment of the application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, which comprises the following steps: first, converting the deterministic scheduling problem into a Markov decision model; the conversion process comprises: establishing the optimization target, the state space, the action space and the reward function of the Markov decision process; then, based on the Markov decision model, constructing a deterministic scheduling algorithm based on a multi-objective hybrid deep Q network and designing a training process to train the algorithm; testing the training result on a random example to form a deterministic scheduling model agent; next, constructing a Markov jump decision process, and converting the event-driven pre-reactive scheduling problem into a Markov jump decision process; the conversion process comprises: definition of the Markov jump decision process and establishment of the state and observation space, the action space and the reward function of the Markov jump decision process; finally, based on the hybrid deep recursive Q network algorithm, constructing a training process, and training the event-driven pre-reactive scheduling agent with the simulation environment; testing the interactive training result on a random example to form an event-driven pre-reactive scheduling agent. The dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning provided by the application realizes the best trade-off in solving precision and solving speed, and the method proposed can handle complex multi-objective scheduling problems with multiple uncertain events.

[0036] The embodiments of the present application will be described in detail below with reference to the drawings. However, those skilled in the art can understand that in the embodiments of the present application, many technical details are proposed in order to enable the reader to better understand the present application. However, the technical solutions claimed by the present application can be implemented even without these technical details and various changes and modifications based on the following embodiments.

[0037] Referring to Figure 1 The embodiments of the present application provide a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, which comprises the following steps:

[0038] Step S101, converting the deterministic scheduling problem into a Markov decision model; the conversion process includes: establishing the optimization goal, the state space, the action space and the reward function of the Markov decision process.

[0039] Step S102, based on the Markov decision model, constructing a deterministic scheduling algorithm based on multi-objective hybrid deep Q network (MOHDQN) and designing a training process to train the algorithm; the training result is tested on a random example to form a deterministic scheduling model agent A1.

[0040] Step S103, constructing a Markov jump decision process, and converting the event-driven pre-reactive scheduling problem into a Markov jump decision process; the conversion process includes: definition of the Markov jump decision process, and establishment of the state and observation space, the action space and the reward function of the Markov jump decision process.

[0041] Step S104, based on the hybrid deep recursive Q network (HDRQN) algorithm, constructing a training process, and training the event-driven pre-reactive scheduling agent with the simulation environment; the interactive training result is tested on a random example to form an event-driven pre-reactive scheduling agent A2.

[0042] The present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, as shown in Figure 2 The present application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, as shown in

[0043] Further, the variables used in all the steps and their explanations are shown in Table 1.

[0044] Table 1 Variables and their related explanations

[0045]

[0046] Further, the calculation formula of the optimization target in the step S101 is as follows:

[0047] Minimize makespan = max (makespan i ) (1)

[0048]

[0049] Wherein, makespan is the maximum value of makespan i of all processes; J i is the i-th workpiece; makespan i is the completion time of the last process of J i ; Tard time represents the total tardiness; D i represents the delay time of J i ; Tard num represents the total number of tardy workpieces; is a variable between 0 and 1, which is used to describe whether makespan i >D i ; U ave represents the average machine utilization; M represents the machine; m is the total number of machines; n is the total number of workpieces; M k represents the k-th machine; U k represents the utilization of M k ; Twork represents the total load of all machines; Twork k represents the load of M k .

[0050] Further, in the step S101, the state space of the Markov decision process is shown in Table 2.

[0051] Table 2 Definition of state space

[0052]

[0053]

[0054] Further, in the step S101, the action space of the Markov decision process is shown in Table 3, which is obtained by combining the machine selection rule and the workpiece selection rule to obtain a 45-dimensional action space.

[0055] Table 3 Definition of action space

[0056]

[0057] Further, the reward function R(s t ,a t ) in the step S101 is defined as the process shown in Algorithm 1.

[0058]

[0059]

[0060] In some embodiments, the deterministic scheduling algorithm based on the multi-objective hybrid deep Q network (MOHDQN) in the step S102 includes a duel network (denoted as Q) and a weight network wherein the Q network includes nine fully connected layers, and the nine fully connected layers include one input layer, six hidden layers and two output layers.

[0061] In order to alleviate the problem of overestimation of Q value, a target network The weight network is used for dynamically allocating weights, including a gated recurrent unit (GRU) and a Softmax activation output layer, and the specific structure is shown in Figure 3 For o optimization objectives, the weight network is mainly used for dynamically allocating weights W = (w1, …, w o ).

[0062] Further, the hyperparameters in the step S102 are set to the values shown in Table 4, and the training method is defined as the process shown in Algorithm 2.

[0063] Table 1 MOHDQN hyperparameter definition

[0064]

[0065]

[0066]

[0067] Further, the Markov jump decision process proposed in the step S103 is defined as an eight-tuple <S, J, E, Z, A, P, r, γ>: state space, jump transition function, dynamic event, observation space, action space, state transition function, reward function and reward decay coefficient, and the specific process is shown in Figure 4 Since the present application adopts model-free reinforcement learning, the above transition function is learned by the algorithm through environment interaction. When the agent starts running, the initial observation Z0 is first observed, and the observation is according to the distribution only when the random event Transition from Z0 to S0. At this time, action a0 is taken and state S0 transitions to observation Z1 according to P(Z1|a0,s0).

[0068] Further, the state and observation space features in step S103 contain 14 dimensions, as shown in Table 5.

[0069] Table 2 State and Observation Space Feature Definitions

[0070]

[0071] Further, the action space in step S103 is composed of 7 rescheduling rules. Before rescheduling, the partial rescheduling is further subdivided according to whether the interrupted operation can be directly inserted into the idle time of the current state of the machine. Specifically, it is necessary to determine whether to reschedule only the interrupted operation or to reschedule all subsequent operations. In order to record workpiece and machine information, machine information list MIL and workpiece information list JIL are constructed. These lists record detailed information such as the start and end times of operations, processing times, assigned workpieces / machines, and scheduled operations on each machine. When a dynamic event occurs at step t, the environment returns the set of interrupted operations DO(t). The idle time of the machine M k and is defined as list ILM k (t). For all processes scheduled to be processed on the machine M k , if , then is put into ILM k (t). The specific determination method is shown in Algorithm 3.

[0072]

[0073] The 7 rescheduling rules are as follows (1)-(7):

[0074] (1) Partial rescheduling rule 1. When there is any insertable rescheduled process, the insertable process is inserted into the earliest machine that can be inserted, and the other processes perform the right shift rescheduling rule. The specific partial rescheduling rule 1 is shown in the process of Algorithm 4.

[0075]

[0076] (2) Partial rescheduling rule 2. When there is any insertable rescheduled process, the insertable process is inserted into the machine with the shortest operation time, and the other processes perform the right shift rescheduling rule. The specific partial rescheduling rule 2 is shown in the process of Algorithm 5.

[0077]

[0078] (3) Hybrid partial rescheduling rule 1. The insertable scheduling procedure is executed by Algorithm 4, and other procedures are partially rescheduled by MOHDQN algorithm in S2. The specific hybrid partial rescheduling rule 1 is shown in the process of Algorithm 6.

[0079]

[0080] (4) Hybrid partial rescheduling rule 2. The insertable scheduling procedure is executed by Algorithm 5, and other procedures are partially rescheduled by MOHDQN algorithm in S2. The specific hybrid partial rescheduling rule 2 is shown in the process of Algorithm 7.

[0081]

[0082]

[0083] (5) Full rescheduling rule. The specific full rescheduling rule is shown in the process of Algorithm 8.

[0084]

[0085] (6) Right shift rescheduling rule.

[0086] (7) Partial rescheduling rule 3. Rescheduling is performed on all affected procedures and their subsequent procedures. The specific partial rescheduling rule 3 is shown in the process of Algorithm 9.

[0087]

[0088] Further, the reward function in step S103 needs to be determined by combining state features and observation features. The state reward function is defined as R(s t ,(z t+1 ,e t+1 )|a t ), and the observation reward function is defined as R(z t ,(s t ,a t )|e t ). Both reward functions follow the same calculation method as shown in Algorithm 10. The comprehensive reward is R(z t ,s t ,e t ,a t ) = αR(s t ,(z t+1 ,e t+1 )|a t ) + βR(z t ,(s t ,a t )|et ). Furthermore, to reduce the time-consuming of algorithm running as low as possible, the single action running time ART(t) is taken into the reward function, then R(z t ,s t ,e t ,a t )=αR(s t ,(z t+1 ,e t+1 )|a t )+βR(z t ,(s t ,a t )|e t )-ART(t)。

[0089]

[0090]

[0091] Further, the HDRQN algorithm in step S104 contains a GRU as the input layer, with a dimension of 28, which is the concatenation of the state and observation space. The hidden layer is a 64-dimensional 3-layer fully connected layer. In the output layer, the output layer dimension of the advantage network is 7, and the output layer dimension of the value network is 1. Finally, it is integrated into a 7-dimensional Q value vector. The hyperparameter definition is shown in Table 6. The structure diagram of the HDRQN algorithm is shown in Figure 5 , and the training method is defined as the process shown in Algorithm 11.

[0092] Table 3 HDRQN hyperparameter definition

[0093]

[0094]

[0095] The data set in the embodiment is randomly generated by a function, and the data set has the features shown in Table 7. In addition, the dynamic event in the embodiment is defined as an external event that causes the original plan to change, and is specifically shown as (1)-(3) as follows:

[0096] (1) Workpiece is late. If and only if S i0 ≤t, the workpiece J i has not arrived and is considered as a dynamic event trigger.

[0097] (2) Workpiece is early. If and only if Ed k (OM i0k -1)<t, the workpiece J i has arrived and is considered as a dynamic event trigger. In the formula, Ed k(x) represents the end time of the xth operation on machine k.

[0098] (3) When the load of the machine exceeds I, and there are multiple processes waiting for processing on the machine, it is considered to be a dynamic event trigger. During maintenance, if there happens to be a process being produced, the production will continue after the inspection is completed.

[0099] Table 4 Random data set generation characteristics

[0100]

[0101] Referring to Figure 6 The embodiment of the present application also provides a dynamic flexible job shop multi-objective scheduling model based on deep reinforcement learning, which adopts the dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning as described in the above embodiment for scheduling, and comprises a Markov decision model module 101, an algorithm construction and training module 102, a Markov jump decision process conversion module 103 and a scheduling intelligent agent training module 104 connected in sequence; the Markov decision model module 101 is used for converting a deterministic scheduling problem into a Markov decision model; the conversion process comprises: establishing an optimization target, a state space, an action space and a reward function of the Markov decision process; the algorithm construction and training module 102 is used for constructing a deterministic scheduling algorithm based on MOHDQN and designing a training process according to the Markov decision model, training the algorithm; testing the training result on a random example to form a deterministic scheduling model intelligent agent; the Markov jump decision process conversion module 103 is used for constructing a Markov jump decision process and converting an event-driven pre- reactive scheduling problem into the Markov jump decision process; the conversion process comprises: defining the Markov jump decision process and establishing a state and observation space, an action space and a reward function of the Markov jump decision process; the scheduling intelligent agent training module 104 is used for constructing a training process according to the HDRQN algorithm and training an event-driven pre- reactive scheduling intelligent agent in interaction with a simulation environment; testing the interactive training result on a random example to form an event-driven pre- reactive scheduling intelligent agent.

[0102] In order to prove the scientificity and accuracy of the method and the model, the present application firstly compares the deterministic scheduling method based on the multi-objective hybrid deep Q network (MOHDQN) with the famous scheduling rules, the composite scheduling rules, the random sample results and the widely recognized multi-objective algorithm NSGA-II. The NSGA-II is tested in 10, 25 and 50 iterations respectively, and the population size is set to 200. The result of random sampling is the average value of 500 random samples. The evaluation indexes are the inverse generational distance (IGD), the hyper volume (HV) and the running time required for each method to generate scheduling. The comparison results are shown in Tables 8 to 12.

[0103] Table 5 MOHDQN vs. simple rule IGD

[0104]

[0105] Table 6 MOHDQN vs. NSGA-II and combined rule IGD

[0106]

[0107]

[0108] IGD evaluates how close the solutions generated by the optimization algorithm are to the theoretical optimal front. The lower the IGD, the closer the solution set generated by the algorithm is to the true Pareto front, and the better the algorithm performs. The algorithm with the lowest IGD values in Tables 8 and 9 are highlighted in bold. The proposed method consistently outperforms the rule-based and composite rule-based methods in all sample sets. MOHDQN can achieve better performance in most cases compared to NSGA-II with only 10 iterations, and is still competitive compared to NSGA-II with 25 iterations. Furthermore, when dealing with large instances, for example, instances involving 50 or more machines and jobs, both MOHDQN and NSGA-II with 50 iterations are able to complete the optimization objective. While random sampling has lower variance, it also has poorer accuracy. MOHDQN exhibits strong stability, consistently providing acceptable Pareto solutions across various samples.

[0109] Table 7 MOHDQN vs. simple rule HV

[0110]

[0111]

[0112] Table 8 MOHDQN vs. NSGA-II and combined rule HV

[0113]

[0114] The volume between the HV measurement and the reference point (usually the worst point in the objective space). The larger the HV value, the larger the area (or volume) occupied by the solution in the objective space, which generally means the better the diversity and quality of the solution. The performance of MOHDQN is more obvious under the HV index than under the IGD index, and this advantage is mainly reflected in large-scale examples. It is worth noting that the HV value of NSGA-II with 50 iterations is very close to MOHDQN. If a larger population size and more iterations are used, it can be reasonably considered that NSGA-II can match or even exceed MOHDQN. However, algorithms that require a large amount of running time are not suitable for fast-changing production environments and are not suitable as basic algorithms for event-driven predictive response scheduling.

[0115] Table 9 MOHDQ algorithm and rule and NSGA-II running time comparison (seconds)

[0116]

[0117] From the above analysis, it can be concluded that the method of the present application is generally superior to the rule-based method and NSGA-II with 10 iterations and 25 iterations in most scenarios. However, it is slightly inferior to NSGA-II with 50 iterations in some scenarios and some indicators. In addition, the MOHDQN method in the present application is mainly used as a generator of scheduling solutions and a basic algorithm for rescheduling, and its running time is as important as its Pareto optimality. Therefore, the present application compares the running time of different algorithms, and the results are shown in Table 11. For problems smaller than 50x40, MOHDQN is the fastest method; beyond this size, it is slower than NSGA-II with 10 iterations. However, for larger problem sizes, the accuracy of NSGA-II with only 10 iterations can no longer be guaranteed. More iterations, while improving accuracy, significantly increase computation time, and NSGA-II is several times or even dozens of times slower than MOHDQN at 50 iterations. Although the rule-based method is also fast, its performance varies greatly between different instances. Therefore, the method proposed in the present application achieves the best trade-off between accuracy and running speed and is more suitable as a basic algorithm for subsequent event-driven pre-reaction scheduling.

[0118] After verifying the effectiveness of the deterministic scheduling method, the present embodiment will continue to evaluate the performance of the event-driven pre-reaction scheduling method HDRQN from the aspects of IGD, HV and running time, and the calculation results are shown in Tables 13 to 15.

[0119] Table 10 HDRQN and rule IGD comparison

[0120]

[0121] In Table 13, the full rescheduling rule, as a method of considering the problem from a global perspective, shows a strong advantage regardless of the size of the example. Although the results of HDRQN are as good as or even better than the full rescheduling rule in some examples, the overall performance of the full rescheduling rule is still better. Among the full rescheduling rule, the proposed HDRQN shows the best and most stable performance, and is superior to other methods in most examples. The right shift rescheduling rule also performs well, although its performance and stability lag behind HDRQN, and this gap further expands when a larger set of examples is tested.

[0122] Table 11 HDRQN vs. Rule HV

[0123]

[0124] In Table 14, the full rescheduling rule still shows a strong performance advantage regardless of the size of the example, but the performance of the full rescheduling rule is at the expense of running efficiency, as shown in Table 16. Among the full rescheduling rule, the accuracy of HDRQN is the best, even exceeding the full rescheduling rule in some examples. Although the partial rescheduling rule 1 has the best stability, its accuracy is poor. The right shift rescheduling rule HV also performs well and can play a certain role in some specific scenarios or industries.

[0125] Table 12 HDRQN vs. Rule Running Time

[0126]

[0127]

[0128] As shown in Table 15, among all the methods, the calculation time of the full rescheduling rule is the longest, the calculation speed of the right-shift rescheduling rule is the fastest, and the HDRQN is between the two. With the increase of the example size, the calculation time of the full rescheduling rule increases exponentially, which can become more serious in a complex actual production environment. Therefore, in combination with Tables 13 to 15, the HDRQN method proposed in the application achieves the best balance between running time and accuracy. However, this does not reduce the value of other rules, and each method has a specific production environment and a more suitable use case. The HDRQN, the full rescheduling rule and the right-shift rescheduling rule each have advantages and disadvantages. The HDRQN provides the best balance between running time and accuracy, the full rescheduling rule achieves the highest accuracy, and the right-shift rescheduling rule is the fastest. For most medium and large enterprises, such as automobile manufacturing, printing industry, etc., dynamic events are frequent, and more efficient scheduling solutions are needed while accuracy cannot be significantly reduced. These industries usually face high production complexity and resource constraints, so the HDRQN is the most suitable choice. While for enterprises with less dynamic events and lower production complexity, such as aerospace or medical device manufacturing enterprises, the full rescheduling rule is the best choice due to its superior accuracy. On the other hand, small enterprises with limited computing power and professional knowledge can benefit more from the right-shift rescheduling rule, as it provides a simpler and faster adjustment to the original production plan.

[0129] According to the technical scheme, the embodiment of the application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning. The method comprises the following steps: firstly, converting a deterministic scheduling problem into a Markov decision model; the conversion process comprises: establishing an optimization target, a state space, an action space and a reward function of the Markov decision process; then, based on the Markov decision model, constructing a MOHDQN-based deterministic scheduling algorithm and designing a training process to train the algorithm; testing the training result on a random example to form a deterministic scheduling model agent; next, constructing a Markov jump decision process and converting an event-driven pre- reactive scheduling problem into the Markov jump decision process; the conversion process comprises: defining the Markov jump decision process and establishing a state and observation space, an action space and a reward function of the Markov jump decision process; finally, based on the HDRQN algorithm, constructing a training process and training an event-driven pre- reactive scheduling agent by interacting with a simulation environment; testing the interactive training result on a random example to form an event-driven pre- reactive scheduling agent.

[0130] The application provides a dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, and realizes event-driven pre-reaction dynamic scheduling. The application converts a deterministic scheduling problem into a Markov decision model, and constructs a deterministic scheduling method based on MOHDQN. Secondly, the constructed deterministic scheduling agent is trained as a scheduling scheme generation method and a rescheduling base method. Finally, a Markov jump decision process is constructed, and the event-driven pre-reaction scheduling process is converted into the Markov jump decision process, which is trained by the HDRQN algorithm to obtain the agent. The method of the application realizes the best trade-off in solving precision and solving speed, and the proposed method can process complex multi-objective scheduling problems with multiple uncertain events.

[0131] Those skilled in the art can understand that each of the above embodiments is a specific embodiment for implementing the application, and in actual application, various changes can be made in form and details without departing from the spirit and scope of the application. Any person skilled in the art can make changes and modifications without departing from the spirit and scope of the application, therefore the protection scope of the application should be limited by the scope defined in the claims.

Claims

1. A dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning, characterized in that, Includes the following steps: Transform the deterministic scheduling problem into a Markov decision model; The transformation process includes: establishing the optimization objective, the state space, action space, and reward function of the Markov decision process; Based on the Markov decision model, a deterministic scheduling algorithm based on a multi-objective hybrid deep Q-network is constructed and a training process is designed to train the algorithm. The training results are then tested on random test cases to form a deterministic scheduling model agent. Construct a Markov jump decision process and transform the event-driven pre-reactive scheduling problem into a Markov jump decision process. The transformation process includes: defining the Markov jump decision process and establishing the state and observation space, action space and reward function of the Markov jump decision process. Based on the hybrid deep recurrent Q-network algorithm, a training process is constructed, and an event-driven pre-reactive scheduling agent is trained interactively with the simulation environment. The interactive training results are tested on random test cases to form an event-driven pre-reactive scheduling agent. The deterministic scheduling algorithm based on a multi-objective hybrid deep Q-network includes a duel network and a weight network; wherein the duel network includes nine fully connected layers, which include one input layer, six hidden layers and two output layers; The weighted network is used to dynamically allocate weights and includes a gated recurrent unit and a Softmax activation output layer. The Markov jump decision process is defined as an octet comprising a state space, a jump transition function, dynamic events, an observation space, an action space, a state transition function, a reward function, and a reward decay coefficient, denoted as: ; The state and observation space of a Markov jump decision process has 14 dimensions: The average machine utilization rate in the scheduling plan is denoted as... ; The number of job delays in the scheduling plan is denoted as ; The delay time in the scheduling plan is denoted as ; In the scheduling plan , recorded as Makespan ; The total machine load in the scheduling plan is denoted as Twork ; The average processing time for all processes is denoted as . ; The time when a dynamic event occurs is denoted as DT ; The average arrival time of the workpiece is denoted as ; The mean time to failure of a machine is denoted as . ; The average idle time of the machine is denoted as ; The average start time of the process is denoted as ; The average completion time of the process is denoted as ; The types of dynamic events; if there are no dynamic events, the value is 0, denoted as . DET ; The number of processes affected is denoted as DON ; The action space of the Markov jump decision process includes 7 rescheduling rules; Before performing a rescheduling, the rescheduling is further subdivided based on whether the interrupt operation can be directly inserted into the machine's current idle time. The reward function of the Markov jump decision process is determined by state characteristics and observation characteristics; The expression for the reward function of the Markov jump decision process is as follows: （6） in, For the state reward function, For the observation reward function; To reduce the algorithm's execution time, the execution time of a single action is... ART ( t If we incorporate this into the reward function, we get: （7） in, ART ( t () represents the execution time of a single action.

2. The multi-objective scheduling method for dynamic flexible job shops based on deep reinforcement learning according to claim 1, characterized in that, The calculation formula for the optimization objective is as follows: （1） （2） （3） （4） （5） in, For all processes The maximum value; For the first i One workpiece; for The completion time of the final process; Indicates the total delay time; Delay time representation The delay time; This indicates the total number of overdue workpieces; A variable between 0 and 1, used to describe whether... ; This represents the average machine utilization rate; M Indicates a machine; m This represents the total number of machines. n This represents the total number of workpieces. M k Indicates the first k One machine; U k express M k Utilization rate; This indicates the total load of all machines; The load indicated.

3. The multi-objective scheduling method for dynamic flexible job shops based on deep reinforcement learning according to claim 1, characterized in that, The parameters of the state space of a Markov decision process include: Average machine utilization Standard deviation of machine utilization Average workpiece process scheduling ratio Average workpiece scheduling rate Standard deviation of average workpiece scheduling rate Estimated workpiece delay rate Current actual workpiece delay rate and the maximum machine utilization rate ; The parameters estimated Current Estimated The difference between the latest delivery date and the latest delivery date is respectively used as Indicated; the estimated delay time and the current actual delay time are respectively represented by _____. The estimated total machine load, the current actual total machine load, and the number of delayed workpieces are represented by ; express.

4. The multi-objective scheduling method for dynamic flexible job shops based on deep reinforcement learning according to claim 1, characterized in that, The action space of the Markov decision process includes machine selection rules and job selection rules; the establishment of the action space includes combining the machine selection rules and job selection rules to obtain a 45-dimensional action space. The machine selection rules include five cases, denoted as: Mrule1, Mrule2, Mrule3, Mrule4, and Mrule5. Rule 1: Select the machine that finishes earliest; Rule 2: Select the machine with the shortest processing time; Rule 3: Select the machine with the least load; Rule 4: Select the machine with the lowest utilization rate; Rule 5: Select the earliest available machine; The workpiece selection rules include nine cases, which are denoted as: Jrule 1, Jrule 2, Jrule 3, Jrule 4, Jrule 5, Jrule 6, Jrule 7, Jrule 8, and Jrule 9. Jrule 1: When there is no workpiece delay, select workpiece i that has been scheduled for workpiece j and satisfies the following conditions. Minimum; responsible for selecting the workpiece with the longest delay; Jrule 2: When there is no workpiece delay, select workpiece i that satisfies Minimum delay; otherwise, select the workpiece with the longest delay. Jrule 3: When there is no workpiece delay, select workpiece i that satisfies... Minimum; Jrule 4: When there is no workpiece delay, select workpiece i that has been scheduled for workpiece j and satisfies the following conditions. Minimum; Jrule 5: Select the workpiece with the shortest processing time for the next process; Jrule 6: Select the workpiece with the longest remaining processing time; Jrule7: Select the workpiece with the earliest delivery date; Jrule 8: When there are no workpiece delays, select the workpiece with the fewest remaining processing steps; otherwise, select the workpiece with the fewest remaining processing steps among the delayed workpieces. Jrule 9: Select the workpiece with the shortest remaining processing time.

5. A dynamic flexible job shop multi-objective scheduling model based on deep reinforcement learning, characterized in that, The scheduling is performed using the dynamic flexible job shop multi-objective scheduling method based on deep reinforcement learning as described in any one of claims 1 to 4, comprising: a Markov decision model module, an algorithm construction and training module, a Markov jump decision process conversion module, and a scheduling agent training module connected in sequence; The Markov decision model module is used to transform the deterministic scheduling problem into a Markov decision model; the transformation process includes: establishing the optimization objective, the state space, action space, and reward function of the Markov decision process; The algorithm construction and training module is used to construct a deterministic scheduling algorithm based on a multi-objective hybrid deep Q-network according to the Markov decision model, design the training process, train the algorithm, and test the training results on random test cases to form a deterministic scheduling model agent. The Markov jump decision process conversion module is used to construct the Markov jump decision process and convert the event-driven pre-reaction scheduling problem into a Markov jump decision process. The conversion process includes: defining the Markov jump decision process and establishing the state and observation space, action space and reward function of the Markov jump decision process. The scheduling agent training module is used to construct the training process according to the hybrid deep recursive Q-network algorithm, and to interactively train the event-driven pre-reactive scheduling agent with the simulation environment; the interactive training results are tested on random cases to form the event-driven pre-reactive scheduling agent.

Images