Flexible job shop scheduling method and system based on deep reinforcement learning and with transportation time

Abstract

The invention discloses a flexible job shop scheduling method with transport time based on deep reinforcement learning, the method adopts a PPO algorithm with a migration network to solve, and the method comprises the following steps: firstly, constructing a flexible job shop scheduling problem example with transport time for model training; secondly, constructing a proper Markov decision model, and converting a target problem into a Markov decision process; thirdly, constructing a proper PPO algorithm agent A1, fully training the agent A1 by using the training data, and optimizing a strategy network and a value network through gradient descent; and finally, constructing a new PPO algorithm agent A2, migrating the strategy network and the value network of the A1 into the A2, performing fine tuning, adding an LSTM network layer and a part of full connection layer, and scheduling a target example by using the A2. The method has good scheduling performance, and the proposed deep reinforcement learning model can efficiently solve flexible job shop scheduling problem examples of various scales with transportation time.

Description

Technical Field

[0001] The present invention belongs to the intersection field of deep reinforcement learning and workshop scheduling, and specifically relates to a flexible job shop scheduling method and system with transportation time based on deep reinforcement learning. Background Art

[0002] As an important part of the manufacturing system, the production workshop directly affects the efficiency and quality of product manufacturing. Production scheduling is one of the important issues that the production workshop needs to consider. Research on workshop intelligent production scheduling is crucial to realize the digitalization, autonomy and intelligence of manufacturing processes, production procedures and systems.

[0003] In manufacturing workshops, handling robots are key equipment to assist machine processing and material transportation, and automated guided vehicles (AGVs) are one of the most common handling robots. AGVs have a high degree of autonomy, flexibility, and strong end-to-end capabilities, and can undertake logistics tasks under multi-variety, small-batch, and large-scale customized production models. Today, AGVs have been widely used in material handling in manufacturing workshops. With the rapid development of AGVs and the improvement of the automation level of manufacturing workshops, workshop scheduling problems have become more and more complex, bringing severe challenges to manufacturing companies. The flexible job shop scheduling problem with transportation time (FJSPT) is an important issue related to improving production efficiency and resource utilization. Compared with the traditional flexible job shop scheduling problem, the machine scheduling in FJSPT is more complicated, including the transportation time that cannot be ignored in actual scheduling, and it is necessary to comprehensively consider the coordination between operations, machines, and AGVs.

[0004] Due to the complexity and diversity of the production environment, traditional scheduling methods, including some priority scheduling rules and meta-heuristic algorithms such as genetic algorithms (GA) and particle swarm algorithms (PSA), face challenges in dealing with complex FJSPT problems. These methods often fail to meet the requirements of time response and exploration and perform poorly in generalization. Deep reinforcement learning (DRL) is an emerging learning method that combines deep learning (DL) and reinforcement learning (RL) techniques and has attracted much attention in recent years. By building an intelligent agent and gradually improving its decision-making ability through interactive learning with the environment, the agent can make great achievements in solving sequential decision-making problems. Deep reinforcement learning builds an adaptive intelligent scheduling model that can better adapt to complex production environments.

[0005] Therefore, developing a more exploratory and adaptable deep reinforcement learning-based flexible job shop scheduling method with transportation time is a technical problem that needs to be solved urgently. Summary of the invention

[0006] The technical problem to be solved by the present invention is to provide a flexible job shop scheduling method and system with transportation time based on deep reinforcement learning, which solves the problem that the traditional job shop scheduling method in the prior art cannot meet the requirements of time response and exploration and performs poorly in generalization.

[0007] The present invention adopts the following technical solutions to solve the above technical problems:

[0008] The flexible job shop scheduling method with transportation time based on deep reinforcement learning includes the following steps:

[0009] Step S1, constructing a flexible job shop scheduling problem example with transportation time, including workpiece information, processing requirements, number of machines, AGV data, distance between machines, etc., which is used as model training data;

[0010] Step S2, constructing a suitable Markov decision model to transform the flexible job shop scheduling problem with transportation time into a Markov decision process; including establishing a state space, an action space, a reward function, etc., and thereby establishing a shop scheduling model;

[0011] Step S3, constructing the first agent of the proximal strategy optimization algorithm, including a strategy network, a value network, a strategy entropy function, an action sampling method, a noise network, and an experience replay pool;

[0012] Step S4: fully train the first agent using the training data, and optimize the policy network and the value network by gradient descent;

[0013] Step S5: construct a second agent of the proximal strategy optimization algorithm, fine-tune the strategy network and value network of the second agent based on the first agent, and add an LSTM network layer and some fully connected layers;

[0014] Step S6: Use the adjusted second agent to schedule the target example.

[0015] In step S2, each state in the state space of the Markov decision process is composed of the artifacts, machines, and AGV features of the scheduling process. When the agent interacts with the environment, environmental features including machine utilization and AGV utilization are extracted from the environment, and state features are calculated based on these environmental features.

[0016] In step S2, the action space of the Markov decision process consists of 6 commonly used workpiece scheduling rules and 4 composite scheduling rules. The composite scheduling rules are composed of ordinary machine scheduling rules and AGV scheduling rules according to their scheduling objectives.

[0017] In step S2, the reward function of the Markov decision process is the execution end time of the action selected in the previous step minus the execution end time of the action selected in this step, that is:

[0018] .

[0019] In step S3, a strategy network and a value network with a linear noise layer are used to improve the generalization performance of the network. The linear noise layer is defined as:

[0020]

[0021] in and are standard weights and bias terms, and is the noise scaling parameter, defining the size of the noise, and is a noise vector, which follows a standard normal distribution. Represents element-wise multiplication.

[0022] In step S3, the action sampling is performed using the decay 𝜖-probability distribution sampling method.

[0023] In step S4, the first agent is fully trained using the training data, and the policy network and the value network are updated by gradient descent during continuous training. The loss function of the value network is:

[0024]

[0025] Due to the addition of the policy entropy function, the policy network loss function is:

[0026]

[0027] in, is the state at time t, is the action at time t, are network parameters, is the strategy entropy function. The larger the entropy value, the stronger the exploratory nature of the algorithm. The calculation formula is:

[0028] .

[0029] In step S5, the backbone of the strategy network and value network of the second agent is derived from the pre-trained first agent, and LSTM is added to ensure its stability and convergence. The calculation formula of LSTM is:

[0030]

[0031]

[0032] .

[0033] A flexible job shop scheduling system with transportation time based on deep reinforcement learning includes a second intelligent agent, a plurality of machines, workpieces, and AGVs, wherein the second intelligent agent applies the flexible job shop scheduling method with transportation time to schedule and control the machines, workpieces, and AGVs.

[0034] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, call all or part of the steps of the method.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. The present invention proposes a new action space configuration, which selects corresponding coordinated composite scheduling rules according to different states of the workshop, which not only narrows the action space and reduces the difficulty of exploration, but also improves the training effect of the algorithm.

[0037] 2. This invention cleverly combines the idea of ​​migration network. First, a PPO agent with a large network scale is constructed, and it is fully trained using examples of various scales, so that the policy network and value network can capture the information in the workshop state and output excellent actions. Then, an A2 agent is constructed to solve specific examples, which greatly reduces the computational cost and improves the generalization of the algorithm.

[0038] 3. This paper proposes a new improved PPO algorithm, which introduces network structures such as LSTM and noise layer to improve the generalization and convergence of the algorithm. It also proposes a new action sampling method, which combines the advantages of greedy sampling and probability distribution sampling to better balance the exploration and utilization of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 This is a flowchart of action selection in an embodiment of the present invention.

[0040] Figure 2 Schematic diagram of the strategy network and value network of agent A2 in an embodiment of the present invention.

[0041] Figure 3 Schematic diagram of the scheduling environment update mechanism in an embodiment of the present invention.

[0042] Figure 4 It is a structural diagram of the overall model in an embodiment of the present invention. DETAILED DESCRIPTION

[0043] The structure and working process of the present invention will be further described below in conjunction with the accompanying drawings.

[0044] In order to break the limitations of the prior art, the present invention provides a flexible job shop scheduling method with transportation time based on deep reinforcement learning. First, the present invention proposes a new action space configuration, which selects corresponding coordinated composite scheduling rules according to different states of the workshop according to different scheduling purposes, reduces the impact of the complexity of FJSPT on the construction of the reinforcement learning environment and improves the scheduling ability of the agent. Secondly, in order to improve the generalization ability of the algorithm and make the model perform well under different problem scales, a reinforcement learning model based on the idea of ​​transfer is proposed. Thirdly, combined with network structures such as Long Short-Term Memory (LSTM) and Noisy Network, a new Proximal Policy Optimization (PPO) method is proposed, and a new action sampling method is proposed to better balance the exploration and utilization of the algorithm and improve the convergence performance of the algorithm.

[0045] To achieve the above objectives, on the one hand, the present invention provides a flexible job shop scheduling method with transportation time based on deep reinforcement learning:

[0046] Step S1, construct a model training example of a flexible job shop scheduling problem with transportation time for subsequent model training, including workpiece information, processing requirements, number of machines, AGV data, distance between machines, etc.

[0047] Step S2: construct a suitable Markov decision model to transform the flexible job shop scheduling problem with transportation time into a Markov decision process, including establishing a state space, an action space, a reward function, etc., and thereby establish a shop scheduling model.

[0048] Step S3: construct a suitable PPO algorithm agent A1, including a policy network, a value network, a policy entropy function, an action sampling method, a noise network, and an experience replay pool.

[0049] Step S4: Use the training data to fully train the agent A1 and optimize the policy network and value network by gradient descent.

[0050] Step S5: Construct a new PPO algorithm agent A2. The policy network and value network of A2 are fine-tuned based on A1, and an LSTM network layer and some fully connected layers are added.

[0051] Step S6: Use the adjusted second agent A2 to schedule the target example.

[0052] Furthermore, in step S1, when generating FJSPT examples, it is necessary to take into account examples of different scales, and the generated example feature information is shown in Table 1 below, where Representative interval Uniform distribution in .

[0053] Table 1. Details of training dataset generation

[0054]

[0055] Furthermore, in step S2, each state in the state space of the Markov decision process is composed of the workpiece, machine, and AGV features of the scheduling process, as shown in Table 2 below.

[0056] Table 2 State characteristics and related variables in FJSPT

[0057]

[0058] When the agent interacts with the environment, it extracts features such as machine utilization and AGV utilization from the environment and calculates state features based on these environmental features. The calculation formulas for state features 1-11 in the table are as follows:

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] Furthermore, in step S2, the action space of the Markov decision process consists of 6 commonly used workpiece scheduling rules and 4 composite scheduling rules. The composite scheduling rules are composed of ordinary machine scheduling rules and AGV scheduling rules according to their scheduling goals, which is beneficial to reducing the scale of the action space, reducing the difficulty of exploration, and improving scheduling efficiency. The 10 scheduling rules of the action space are shown in Table 3 below.

[0071] Table 3 Action space scheduling rules

[0072]

[0073] in The specific implementation process is shown in Table 4.

[0074] Table 4 Specific implementation process

[0075] Furthermore, in step S2, the reward function of the Markov decision process is the execution end time of the action selected in the previous step (i.e., step) minus the execution end time of the action selected in this step, that is:

[0076]

[0077] This reward function is an equivalent reward and meets the principles of clarity, sparseness, timeliness, and learnability. There are two advantages to adopting such a cumulative reward: ① The problem of too sparse rewards will not occur; ② The cumulative reward is the negative value of makespan, and maximizing the cumulative reward is completely consistent with the goal of workshop scheduling.

[0078] Furthermore, in step S3, a strategy network and a value network with a linear noise layer are used to improve the generalization performance of the network, and the linear noise layer is defined as:

[0079]

[0080] in and are standard weights and bias terms, and is the noise scaling parameter, defining the size of the noise, and is a noise vector, which follows a standard normal distribution. Represents element-wise multiplication.

[0081] Furthermore, in step S3, attenuation is used -Probability distribution sampling method is used for action sampling. When sampling, the action with the highest Q value is directly selected with a certain probability, and the action is randomly selected from all actions according to the probability distribution formed by the Q value with a certain probability. This ensures the convergence performance of the model and can take into account a certain amount of exploration. -The specific implementation of the probability distribution sampling method is shown in Table 5 below.

[0082] Table 5 Attenuation - Probability distribution sampling

[0083] An exponential decay method is used to The update is smoother and the number of iterations is How is the value of ? The value of is always The number of nodes increases monotonically, maintaining a dynamic balance between exploration and utilization.

[0084] Furthermore, in step S4, the agent A1 is fully trained using the training data, and the policy network and the value network are updated by gradient descent during continuous training. The loss function of the value network is:

[0085]

[0086] Due to the addition of the policy entropy function, the policy network loss function is:

[0087]

[0088] in is the strategy entropy function. The larger the entropy value, the stronger the exploratory nature of the algorithm. The calculation formula is:

[0089]

[0090] Furthermore, in step S5, the backbone of the policy network and value network of A2 is derived from the pre-trained A1 agent, and LSTM is added to ensure its stability and convergence. The calculation formula of LSTM is:

[0091]

[0092]

[0093]

[0094] LSTM is added to the strategy network and value network of PPO, and the superiority of LSTM in processing sequence data is used to fully extract important information in the input state. The input state is increased from the original one time step information to several time steps information, making the training process more efficient and easy to converge.

[0095] Specific embodiments, such as Figures 1 to 4 As shown,

[0096] In the following detailed description, many specific details are set forth to provide a more thorough understanding of the present invention. However, it is obvious to those skilled in the art that the well-known control algorithms are not shown in detail to avoid blurring the main idea of ​​the present invention; and the Markov decision process, PPO algorithm, LSTM network, noise network and other technologies involved in the following effect embodiments are prior arts that can be retrieved.

[0097] See also Figure 1-4 , a flexible job shop scheduling method with transportation time based on deep reinforcement learning is applied, and the algorithm parameters are designed as shown in Table 6.

[0098] Table 6 Algorithm hyperparameter settings

[0099]

[0100] This example divides the data set into two parts: a training data set and a test data set. The training data set is randomly generated by python. In order to fully train the agent, the generation of the training data set follows several random distributions with a large span. The details of the training data set generation are shown in Table 1. A large amount of training data is used to train the agent A1 so that it has sufficiently strong generalization performance. The test data set of the present invention contains 10 small FJSPT examples (FJSPT1-10) and 3 large FJSPT examples (MFJS08, MFJS09 and MFJS10). The above 13 examples all use 2 AGVs and 8 machines, and the layout of the machines and the transportation time of the AGVs between the machines are consistent. The advantage of using these 13 benchmark instances as test data sets is that there are instances of different scales, which can fully demonstrate the generalization performance of the method of the present invention on problems of different scales.

[0101] In order to verify the effectiveness and feasibility of the algorithm of the present invention, the results of the algorithm of the present invention are compared with the results of some other commonly used baseline algorithms in this section, mainly including some priority scheduling rules, such as:

[0102] (1) ; (2) ; (3) ; (4) ; (5) ; (6) etc., as well as some advanced solutions to the FJSPT problem proposed by some scholars, including an improved genetic algorithm (PGA), a dual population collaborative genetic algorithm (DCGA) and the DDQN method. The performance comparison of different baselines is shown in Table 7.

[0103] In Table 7, the reference optimal solution comes from the mixed integer linear programming model. Due to the complexity of MFJS08-10, linear programming cannot give the optimal solution. Instead, the best result of the above method is used as the reference optimal solution. The result of DDQN is the average of 20 independent simulations. GAP is used to compare the scheduling performance of the above different methods, and its calculation formula is:

[0104]

[0105] in is the makespan result obtained by the corresponding algorithm, is the reference solution of the example.

[0106] In terms of scheduling performance, the performance of different PDR methods varies significantly, especially PDR5-6 is significantly better than PDR1-4. This phenomenon may be due to the fact that PDR5-6 adopts The scheduling rules are related to the scheduling rules, which ensure that the processing can start as early as possible by selecting appropriate machines and AGVs, so that the work can be completed in a shorter time. Although the scheduling performance of PDR5-6 is better, the algorithm proposed in the present invention still shows better scheduling effect in all test cases, which shows the effectiveness of the algorithm proposed in the present invention.

[0107] Compared with the most advanced deep reinforcement learning algorithm DDQN, the method of the present invention achieves better results in most cases, which further proves the superiority of the algorithm of the present invention. In addition, although there is still a gap with the latest meta-heuristic algorithm DCGA, the table data shows that the performance of the method of the present invention in small-scale instances is comparable to that of the more advanced meta-heuristic algorithm PGA, and the performance is relatively close.

[0108] Table 7 Performance comparison of different methods

[0109] For large-scale examples (such as MFJS08-MFJS10), the scheduling effect of PDR1-4 is poor, and each GAP value exceeds 200%, and some even exceed 300%. This shows that as the complexity of the example increases, the effect of the PDR method gradually decreases, especially when dealing with more complex examples. In contrast, the method proposed in the present invention shows strong exploratoryness when dealing with large-scale examples, especially in examples such as MFJS08-MFJS10, its scheduling results are better than the PGA algorithm, showing the advantages of algorithms based on deep reinforcement learning in large-scale problems. Although the performance of the method of the present invention is slightly inferior to the meta-heuristic algorithm PGA in small-scale problems, when facing large-scale examples, the deep reinforcement learning method significantly improves the scheduling effect through a more effective search strategy. The deep reinforcement learning algorithm can improve the search performance in a limited time, thereby exploring the solution space more comprehensively. In summary, although the advantages of the method of the present invention are not obvious in small-scale problems, in large-scale problems, as the solution space expands, it shows stronger adaptability and advantages.

[0110] The present invention also discloses a flexible job shop scheduling system with transportation time based on deep reinforcement learning, including a second intelligent agent, a number of machines, workpieces, and AGVs, wherein the second intelligent agent applies the flexible job shop scheduling method with transportation time to schedule and control the machines, workpieces, and AGVs.

[0111] A computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, call all or part of the steps of the method.

[0112] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0113] Those skilled in the art should understand that those skilled in the art can implement variations by combining the prior art and the above embodiments, which will not be described in detail here. Such variations do not affect the essential content of the present invention, and will not be described in detail here.

[0114] The above describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the above-mentioned specific embodiments, and the devices and structures that are not described in detail should be understood to be implemented in a common manner in the art; any technician familiar with the art can use the above-disclosed methods and technical contents to make many possible changes and modifications to the technical solutions of the present invention without departing from the scope of the technical solutions of the present invention, or modify them into equivalent embodiments of equivalent changes, which does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solutions of the present invention are still within the scope of protection of the technical solutions of the present invention.

Claims

1. A flexible job shop scheduling method with transportation time based on deep reinforcement learning, characterized by: The steps include: Step S1, constructing a flexible job shop scheduling problem example with transportation time, including workpiece information, processing requirements, number of machines, AGV data, and distance between machines, which are used as model training data; Step S2: construct a suitable Markov decision model to transform the flexible job shop scheduling problem with transportation time into a Markov decision process; This includes establishing state space, action space, and reward function, and using them to establish a workshop scheduling model; Step S3, constructing the first agent of the proximal strategy optimization algorithm, including a strategy network, a value network, a strategy entropy function, an action sampling method, a noise network and an experience replay pool; Step S4: fully train the first agent using the training data, and optimize the policy network and the value network by gradient descent; Step S5: construct a second agent of the proximal strategy optimization algorithm, fine-tune the strategy network and value network of the second agent based on the first agent, and add an LSTM network layer and some fully connected layers; Step S6: Use the adjusted second agent to schedule the target example.

2. The method for flexible job shop scheduling with transportation time based on deep reinforcement learning according to claim 1 is characterized in that: In step S2, each state in the state space of the Markov decision process is composed of the artifacts, machines, and AGV features of the scheduling process. When the agent interacts with the environment, environmental features including machine utilization and AGV utilization are extracted from the environment, and state features are calculated based on these environmental features.

3. The flexible job shop scheduling method with transportation time based on deep reinforcement learning according to claim 1 is characterized in that: In step S2, the action space of the Markov decision process consists of 6 commonly used workpiece scheduling rules and 4 composite scheduling rules. The composite scheduling rules are composed of ordinary machine scheduling rules and AGV scheduling rules according to their scheduling objectives.

4. The flexible job shop scheduling method with transportation time based on deep reinforcement learning according to claim 1 is characterized in that: In step S2, the reward function of the Markov decision process is the execution end time of the action selected in the previous step minus the execution end time of the action selected in this step, that is: 。 5. The method for flexible job shop scheduling with transportation time based on deep reinforcement learning according to claim 1 is characterized in that: In step S3, a strategy network and a value network with a linear noise layer are used to improve the generalization performance of the network. The linear noise layer is defined as: in and are standard weights and bias terms, and is the noise scaling parameter, defining the size of the noise, and is a noise vector, which follows a standard normal distribution. Represents element-wise multiplication.

6. The flexible job shop scheduling method with transportation time based on deep reinforcement learning according to claim 1 is characterized in that: In step S3, the action sampling is performed using the decay 𝜖-probability distribution sampling method.

7. The method for flexible job shop scheduling with transportation time based on deep reinforcement learning according to claim 1 is characterized in that: In step S4, the first agent is fully trained using the training data, and the policy network and the value network are updated by gradient descent during continuous training. The loss function of the value network is: Due to the addition of the policy entropy function, the policy network loss function is: in, is the state at time t, is the action at time t, are network parameters, is the strategy entropy function. The larger the entropy value, the stronger the exploratory nature of the algorithm. The calculation formula is: 。 8. The method for flexible job shop scheduling with transportation time based on deep reinforcement learning according to claim 1 is characterized in that: In step S5, the backbone of the strategy network and value network of the second agent is derived from the pre-trained first agent, and LSTM is added to ensure its stability and convergence. The calculation formula of LSTM is: 。 9. A flexible job shop scheduling system with transportation time based on deep reinforcement learning, characterized by: It includes a second intelligent agent, several machines, workpieces, and AGVs, wherein the second intelligent agent applies the flexible job shop scheduling method with transportation time as described in any one of claims 1 to 8 to schedule and control the machines, workpieces, and AGVs.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by a processor, all or part of the steps of the method described in any one of claims 1 to 8 are called.

Images