Flexible job shop scheduling method and system based on deep reinforcement learning with transportation time

Abstract

The application discloses a flexible job shop scheduling method with transportation time based on deep reinforcement learning, and adopts a PPO algorithm with a migration network to solve the method, and the method comprises the following steps: firstly, a flexible job shop scheduling problem example with transportation time is constructed to be used for model training; secondly, a suitable Markov decision model is constructed, and the target problem is converted into a Markov decision process; thirdly, a suitable PPO algorithm intelligent agent A1 is constructed, the intelligent agent A1 is fully trained by using training data, and the policy network and the value network are optimized through gradient descent; finally, a new PPO algorithm intelligent agent A2 is constructed, the policy network and the value network of A1 are migrated to A2 and fine-tuned, an LSTM network layer and part of the full connection layer are added, and A2 is used to schedule the target example. The application has good scheduling performance, and the deep reinforcement learning model can efficiently solve flexible job shop scheduling problem examples with transportation time of various scales.

Description

Technical Field

[0001] This invention belongs to the interdisciplinary field of deep reinforcement learning and job shop scheduling, specifically involving a flexible job shop scheduling method and system based on deep reinforcement learning with transportation time. Background Technology

[0002] As a crucial component of the manufacturing system, the production workshop directly impacts product manufacturing efficiency and quality. Production scheduling is one of the key issues that production workshops need to consider. Researching intelligent production scheduling in the workshop is essential for achieving the digitalization, autonomy, and intelligence of manufacturing processes, production procedures, and systems.

[0003] In manufacturing workshops, material handling robots are key equipment assisting in machining and material transport, and Automated Guided Vehicles (AGVs) are among the most common types of material handling robots. AGVs possess high autonomy, flexibility, and strong end-to-end capabilities, enabling them to handle logistics tasks in multi-variety, small-batch, and large-scale customized production models. Today, AGVs are widely used in material handling in manufacturing workshops. With the rapid development of AGVs and the increasing automation level of manufacturing workshops, workshop scheduling problems have become increasingly complex, posing significant challenges to manufacturing enterprises. The Flexible Job Shop Scheduling Problem with Transport Time (FJSPT) is a crucial issue related to improving production efficiency and resource utilization. Compared to traditional flexible job shop scheduling problems, machine scheduling in FJSPT is more complex, incorporating the significant transport time inherent in actual scheduling, and requiring comprehensive consideration of the coordination between operations, machines, and AGVs.

[0004] Due to the complexity and diversity of production environments, traditional scheduling methods, including priority scheduling rules and metaheuristic algorithms such as Genetic Algorithms (GA) and Particle Swarm Optimization (PSA), face challenges in handling complex FJSPT problems. These methods often fail to meet the requirements of time response and exploration, and perform poorly in generalization. Deep Reinforcement Learning (DRL), an emerging learning method combining Deep Learning (DL) and Reinforcement Learning (RL) techniques, has attracted much attention in recent years. By constructing intelligent agents and gradually improving their decision-making abilities through interaction with the environment, these agents can achieve great success in solving sequential decision-making problems. Deep Reinforcement Learning constructs adaptive intelligent scheduling models that can better adapt to complex production environments.

[0005] Therefore, developing a more exploratory and adaptive flexible job shop scheduling method based on deep reinforcement learning with transportation time is a technical problem that urgently needs to be solved. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a flexible job shop scheduling method and system based on deep reinforcement learning with transportation time, which solves the problem that the traditional job shop scheduling method in the prior art cannot meet the requirements of time response and exploration, and performs poorly in generalization.

[0007] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0008] A flexible job shop scheduling method based on deep reinforcement learning with transportation time includes the following steps:

[0009] Step S1: Construct a case study of the flexible job shop scheduling problem with transportation time; including workpiece information, processing requirements, number of machines, AGV data, distance between machines, etc., to be used as model training data;

[0010] Step S2: Construct a suitable Markov decision model to transform the flexible job shop scheduling problem with transportation time into a Markov decision process; this includes establishing the state space, action space, reward function, etc., and using these to build the job shop scheduling model.

[0011] Step S3: Construct the first agent of the near-end policy optimization algorithm, including policy network, value network, policy entropy function, action sampling method, noise network and experience replay pool, etc.

[0012] Step S4: Train the first agent fully using training data, and optimize the policy network and value network through gradient descent;

[0013] Step S5: Construct a second agent for the near-end policy optimization algorithm. Based on the first agent, fine-tune the policy network and value network of the second agent, and add an LSTM network layer and some fully connected layers.

[0014] Step S6: Use the adjusted second agent to schedule the target computation.

[0015] In step S2, each state in the state space of the Markov decision process consists of the features of the workpiece, machine, and AGV in the scheduling process. When the agent interacts with the environment, it extracts environmental features including machine utilization and AGV utilization from the environment and calculates state features based on these environmental features.

[0016] In step S2, the action space of the Markov decision process consists of 6 commonly used workpiece scheduling rules and 4 composite scheduling rules. The composite scheduling rules are composed of ordinary machine scheduling rules and AGV scheduling rules combined according to their scheduling objectives.

[0017] In step S2, the reward function of the Markov decision process is calculated by subtracting the execution end time of the action selected in the current step from the execution end time of the action selected in the previous step, i.e.:

[0018] .

[0019] In step S3, a policy network and a value network with a linear noise layer are used to improve the network's generalization performance. The linear noise layer is defined as follows:

[0020]

[0021] in and These are the standard weights and bias terms. and This is a noise scaling parameter that defines the magnitude of the noise. and It is a noise vector that follows a standard normal distribution. This indicates element-wise multiplication.

[0022] In step S3, the attenuation 𝜖-probability distribution sampling method is used for action sampling.

[0023] In step S4, the first agent is fully trained using training data. During continuous training, the policy network and value network are updated using gradient descent. The loss function of the value network is:

[0024]

[0025] With the addition of the policy entropy function, the policy network loss function is:

[0026]

[0027] in, The state at time t, For the action at time t, For network parameters, Let be the policy entropy function. A larger entropy value indicates a stronger exploratory nature of the algorithm. Its calculation formula is:

[0028] .

[0029] In step S5, the backbone of the policy network and value network of the second agent originates from the pre-trained first agent, and then LSTM is added to ensure its stability and convergence. The calculation formula of LSTM is:

[0030]

[0031]

[0032] .

[0033] A flexible job shop scheduling system based on deep reinforcement learning with transportation time includes a second intelligent agent, several machines, workpieces, and AGVs. The second intelligent agent applies the flexible job shop scheduling method with transportation time to schedule and control the machines, workpieces, and AGVs.

[0034] A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, invoke all or part of the steps of the method.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] 1. This invention proposes a new action space configuration, which selects corresponding coordinated composite scheduling rules according to different workshop states, thereby reducing the action space, lowering the exploration difficulty, and improving the training effect of the algorithm.

[0037] 2. This invention cleverly combines the idea of ​​transfer networks. First, a large-scale PPO agent is constructed and trained extensively using computational examples of various sizes, enabling the policy network and value network to capture information from the workshop state and output optimal actions. Then, an A2 agent is constructed to solve specific computational examples, significantly reducing computational costs and improving the algorithm's generalization ability.

[0038] 3. This invention proposes a novel improved PPO algorithm, introducing network structures such as LSTM and a noise layer to enhance the algorithm's generalization and convergence. It also proposes a new action sampling method that combines the advantages of greedy sampling and probability distribution sampling, achieving a better balance between algorithm exploration and utilization. Attached Figure Description

[0039] Figure 1 This is a flowchart of the action selection process in an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of the agent A2 policy network and value network in an embodiment of the present invention.

[0041] Figure 3 This is a schematic diagram of the scheduling environment update mechanism in an embodiment of the present invention.

[0042] Figure 4 This is a structural diagram of the overall model in an embodiment of the present invention. Detailed Implementation

[0043] The structure and working process of the present invention will be further described below with reference to the accompanying drawings.

[0044] To overcome the limitations of existing technologies, this invention provides a flexible job shop scheduling method with transport time based on deep reinforcement learning. First, this invention proposes a novel action space configuration, selecting corresponding coordinated composite scheduling rules based on different shop states according to different scheduling objectives, reducing the impact of FJSPT complexity on reinforcement learning environment construction and improving agent scheduling capabilities. Second, to improve the algorithm's generalization ability and ensure good performance across different problem scales, a reinforcement learning model based on transfer learning is proposed. Third, combining network structures such as Long Short-Term Memory (LSTM) and noisy networks, a novel Proximal Policy Optimization (PPO) method is proposed, along with a new action sampling method, to better balance the algorithm's exploration and utilization, thereby improving convergence performance.

[0045] To achieve the above objectives, on the one hand, this invention provides a flexible job shop scheduling method with transportation time based on deep reinforcement learning:

[0046] Step S1: Construct a case study of the flexible job shop scheduling problem with transportation time for model training, which will be used for subsequent model training. This case study includes workpiece information, processing requirements, number of machines, AGV data, and distance between machines.

[0047] Step S2: Construct a suitable Markov decision model to transform the flexible job shop scheduling problem with transportation time into a Markov decision process, including establishing the state space, action space, reward function, etc., and then establishing the job shop scheduling model based on it.

[0048] Step S3: Construct a suitable PPO algorithm agent A1, including a policy network, a value network, a policy entropy function, an action sampling method, a noise network, and an experience replay pool.

[0049] Step S4: Train agent A1 fully using training data, and optimize the policy network and value network through gradient descent.

[0050] Step S5: Construct a new PPO algorithm agent A2. The policy network and value network of A2 are fine-tuned based on A1, and LSTM network layers and some fully connected layers are added.

[0051] Step S6: Use the adjusted second agent A2 to schedule the target computation.

[0052] Furthermore, in step S1, when generating FJSPT cases, it is necessary to consider cases of different sizes. The characteristic information of the generated cases is shown in Table 1 below, where... Representative interval The uniform distribution within.

[0053] Table 1. Details of training dataset generation

[0054]

[0055] Furthermore, in step S2, each state in the state space of the Markov decision process is composed of the workpiece, machine, and AGV features of the scheduling process, as shown in Table 2 below.

[0056] Table 2. State characteristics and related variables in FJSPT

[0057]

[0058] When an intelligent agent interacts with its environment, it extracts features such as machine utilization and AGV utilization from the environment and calculates state features based on these environmental features. The calculation formulas for state features 1-11 in the table are as follows:

[0059]

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070] Furthermore, in step S2, the action space of the Markov decision process consists of six commonly used workpiece scheduling rules and four composite scheduling rules. The composite scheduling rules are combinations of ordinary machine scheduling rules and AGV scheduling rules based on their scheduling objectives, which helps reduce the size of the action space, lower the exploration difficulty, and improve scheduling efficiency. The ten scheduling rules of the action space are shown in Table 3 below.

[0071] Table 3 Action Space Scheduling Rules

[0072]

[0073] in The specific implementation process is shown in Table 4.

[0074] Table 4 Specific implementation process

[0075] Furthermore, in step S2, the reward function of the Markov decision process is calculated by subtracting the execution end time of the action selected in the current step from the execution end time of the action selected in the previous step (i.e., step S2), that is:

[0076]

[0077] The reward function is an equivalent reward and satisfies the principles of being explicit, sparse, timely, and learnable. Adopting such a cumulative reward has two advantages: ① It avoids the problem of excessively sparse rewards; ② The cumulative reward is a negative value of makespan, and maximizing the cumulative reward is completely consistent with the goal of shop floor scheduling.

[0078] Furthermore, in step S3, a policy network and a value network with a linear noise layer are used to improve the network's generalization performance. The linear noise layer is defined as:

[0079]

[0080] in and These are the standard weights and bias terms. and This is a noise scaling parameter that defines the magnitude of the noise. and It is a noise vector that follows a standard normal distribution. This indicates element-wise multiplication.

[0081] Furthermore, in step S3, attenuation is used. - Action sampling is performed using a probability distribution sampling method. During sampling, the action with the highest Q-value is selected directly with a certain probability, while another action is randomly selected from all actions based on a probability distribution formed by the Q-values ​​with a certain probability. This ensures the model's convergence performance while also allowing for some exploration. (Fallendown) The specific implementation methods of the probability distribution sampling method are shown in Table 5 below.

[0082] Table 5 Attenuation -Probability distribution sampling

[0083] It employs an exponential decay method. The advantage of performing updates in this way is that the updates are relatively smooth, regardless of the number of iterations. What are the possible values ​​for ? The value is always in The levels monotonically increase, maintaining a dynamic balance between exploration and utilization.

[0084] Furthermore, in step S4, the agent A1 is fully trained using training data, and the policy network and value network are updated through gradient descent during continuous training. The loss function of the value network is:

[0085]

[0086] With the addition of the policy entropy function, the policy network loss function is:

[0087]

[0088] in Let be the policy entropy function. A larger entropy value indicates a stronger exploratory nature of the algorithm. Its calculation formula is:

[0089]

[0090] Furthermore, in step S5, the backbone of the policy network and value network of A2 originates from the pre-trained A1 agent, and an LSTM is added to ensure its stability and convergence. The calculation formula for LSTM is:

[0091]

[0092]

[0093]

[0094] By incorporating LSTM into the policy and value networks of PPO, the superiority of LSTM in processing sequence data is fully utilized to extract important information from the input state. The information from the input state is increased from one time step to several time steps, making the training process more efficient and easier to converge.

[0095] Specific embodiments, such as Figures 1 to 4 As shown,

[0096] In the following detailed description, numerous specific details are set forth to provide a more thorough understanding of the invention. However, it will be apparent to those skilled in the art that known control algorithms are not shown in detail to avoid obscuring the gist of the invention; and that the Markov decision processes, PPO algorithms, LSTM networks, noisy networks, and other techniques involved in the following effect examples are readily available prior art.

[0097] See Figure 1-4 A flexible job shop scheduling method with transportation time based on deep reinforcement learning is applied. The algorithm parameters are shown in Table 6.

[0098] Table 6 Algorithm Hyperparameter Settings

[0099]

[0100] This example divides the dataset into two parts: a training dataset and a test dataset. The training dataset is randomly generated by Python. To fully train the agent, the training dataset is generated following several random distributions with large spans. Details of the training dataset generation are shown in Table 1. A large amount of training data is used to train the agent A1 to achieve sufficiently strong generalization performance. The test dataset of this invention contains 10 small FJSPT examples (FJSPT1-10) and 3 large FJSPT examples (MFJS08, MFJS09, and MFJS10). All 13 examples use 2 AGVs and 8 machines, with consistent machine layout and AGV transport time between machines. The advantage of using these 13 benchmark examples as the test dataset is that they represent examples of different sizes, allowing for a comprehensive demonstration of the generalization performance of the method of this invention on problems of different scales.

[0101] To verify the effectiveness and feasibility of the algorithm of this invention, the results of the algorithm of this invention are compared with the results of some other commonly used baseline algorithms in this section, mainly including some priority scheduling rules, such as:

[0102] (1) (2) (3) (4) (5) (6) Other methods, including an improved genetic algorithm (PGA), a two-population cooperative genetic algorithm (DCGA), and the DDQN method, have been proposed by scholars to address the FJSPT problem. Performance comparisons of different baselines are shown in Table 7.

[0103] In Table 7, the reference optimal solution comes from the mixed-integer linear programming model. Due to the complexity of MFJS08-10, linear programming cannot provide an optimal solution. Instead, the best result from the above method is used as the reference optimal solution. The result of DDQN is the average of 20 independent simulations. GAP is used to compare the scheduling performance of the different methods mentioned above, and its calculation formula is as follows:

[0104]

[0105] in This is the makespan result obtained from the corresponding algorithm. This is the reference solution for the example.

[0106] In terms of scheduling performance, the different PDR methods showed significant differences, especially PDR5-6, which was significantly better than PDR1-4. This phenomenon may be due to the fact that PDR5-6 adopted... The scheduling rules are relevant; these rules ensure that processing can begin as early as possible by selecting appropriate machines and AGVs, thus enabling the job to be completed in a shorter time. Although PDR5-6 performed well in scheduling, the algorithm proposed in this invention still showed superior scheduling performance in all test cases, demonstrating the effectiveness of the proposed algorithm.

[0107] Compared to the state-of-the-art deep reinforcement learning algorithm DDQN, the method of this invention achieves better results in most cases, further demonstrating the superiority of the algorithm. Furthermore, although there is still a gap compared to the latest metaheuristic algorithm DCGA, the data in the table show that the performance of the method of this invention on small-scale instances is comparable to that of the more advanced metaheuristic algorithm PGA, showing a close match.

[0108] Table 7 Performance Comparison of Different Methods

[0109] For large-scale examples (such as MFJS08-MFJS10), PDR1-4 exhibit poor scheduling performance, with each gap exceeding 200%, and some even exceeding 300%. This indicates that the effectiveness of the PDR method gradually decreases as the complexity of the example increases, especially when dealing with more complex examples. In contrast, the method proposed in this invention demonstrates strong exploratory capabilities when handling large-scale instances, particularly in examples like MFJS08-MFJS10, where its scheduling results outperform the PGA algorithm, showcasing the advantages of deep reinforcement learning-based algorithms for large-scale problems. Although the performance of the method in this invention is slightly inferior to the metaheuristic algorithm PGA in small-scale problems, the deep reinforcement learning method significantly improves scheduling performance through a more effective search strategy when facing large-scale instances. Deep reinforcement learning algorithms can improve search performance within a finite time, thereby exploring the solution space more comprehensively. In summary, although the advantage of the method in this invention is not significant in small-scale problems, it exhibits stronger adaptability and advantages in large-scale problems as the solution space expands.

[0110] The present invention also discloses a flexible job shop scheduling system with transportation time based on deep reinforcement learning, including a second intelligent agent, several machines, workpieces, and AGVs, wherein the second intelligent agent applies the flexible job shop scheduling method with transportation time to schedule and control the machines, workpieces, and AGVs.

[0111] A computer-readable storage medium storing computer-readable instructions that, when executed by a processor, invoke all or part of the steps of the method.

[0112] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0113] Those skilled in the art should understand that variations can be implemented by combining existing technology with the above embodiments, which will not be elaborated here. Such variations do not affect the essence of the present invention, and will not be elaborated here either.

[0114] The preferred embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and the devices and structures not described in detail should be understood as being implemented in a conventional manner in the art. Any person skilled in the art can make many possible variations and modifications to the technical solutions of the present invention using the methods and techniques disclosed above, or modify them into equivalent embodiments with equivalent changes, without departing from the scope of the present invention. This does not affect the essential content of the present invention. Therefore, any simple modifications, equivalent changes, and modifications made to the above embodiments based on the technical essence of the present invention without departing from the content of the present invention's technical solutions still fall within the protection scope of the present invention.

Claims

1. A flexible job shop scheduling method based on deep reinforcement learning with transportation time, characterized in that: Includes the following steps: Step S1: Construct a case study of the flexible job shop scheduling problem with transportation time; including workpiece information, processing requirements, number of machines, AGV data, and distance between machines, to be used as model training data; Step S2: Construct a suitable Markov decision model to transform the flexible job shop scheduling problem with transportation time into a Markov decision process; This includes establishing the state space, action space, and reward function, and using these to build a workshop scheduling model; Step S3: Construct the first agent of the near-end policy optimization algorithm, including policy network, value network, policy entropy function, action sampling method, noise network and experience replay pool; Use attenuation - A probability distribution sampling method is used for action sampling. During sampling, the action with the highest Q value is directly selected with a certain probability, while another action is randomly selected from all actions based on a probability distribution formed by the Q values. An exponential decay method is then used to further sample actions. Updates are performed; action sampling is performed using the decaying 𝜖-probability distribution sampling method, maintaining a dynamic balance between exploration and exploitation. The specific method of decaying 𝜖-probability distribution sampling is as follows: (1) Get the current state Action probability distribution output by the lower policy network ; (2) Generate random numbers and the random number With threshold Comparison, among which For greedy parameters; (3) If Then, a greedy exploitation strategy is executed, directly selecting the aforementioned action probability distribution. The action with the highest probability value ; (4) If Then, a guided exploration strategy is executed, directly based on the aforementioned action probability distribution. Sampling is performed according to a probability distribution, rather than uniform random sampling; (5) After each sampling iteration, the greedy parameter The update is performed using the exponential decay formula, which is: ;in To greedily access the upper bound of the parameter, This is the lower bound of the greedy parameter. For the number of samples, The decay rate is a greedy parameter. Step S4: Train the first agent fully using a large amount of training data, and optimize the policy network and value network through gradient descent to make them have sufficiently strong generalization performance. Step S5: Construct the second agent of the proximal policy optimization algorithm. The backbone of the policy network and value network of the second agent comes from the pre-trained first agent. Based on the first agent, the policy network and value network of the second agent are fine-tuned, and LSTM network layers and some fully connected layers are added to ensure its stability and convergence. Step S6: Use the adjusted second agent to schedule the target computation.

2. The flexible job shop scheduling method based on deep reinforcement learning with transportation time as described in claim 1, characterized in that: In step S2, each state in the state space of the Markov decision process is composed of the workpiece, machine, and AGV features of the scheduling process. When the agent interacts with the environment, it extracts environmental features from the environment, including machine average utilization rate, machine utilization rate standard deviation, AGV average utilization rate, AGV utilization rate standard deviation, AGV maximum utilization rate, average process completion rate, mean process completion rate of each workpiece, standard deviation of process completion rate of each workpiece, expected processing completion time, and current processing time. Based on these environmental features, 11 types of state features are calculated.

3. The flexible job shop scheduling method based on deep reinforcement learning with transportation time as described in claim 1, characterized in that: In step S2, the action space of the Markov decision process consists of 6 commonly used workpiece scheduling rules and 4 composite scheduling rules. The composite scheduling rules are composed of ordinary machine scheduling rules and AGV scheduling rules combined according to their scheduling objectives. The composite scheduling rules specifically include: Select the workpiece with the shortest remaining processing time, select the workpiece with the longest remaining processing time, select the workpiece with the shortest processing time for the next process, select the workpiece with the longest processing time for the next process, select the workpiece with the fewest remaining processes, select the workpiece with the most remaining processes, select the machine and AGV that minimizes the workpiece receiving time and processing time, select the machine and AGV that minimizes the workpiece delivery time, select the machine and AGV with the lowest utilization rate, and select the machine and AGV that allows the workpiece to start processing earliest.

4. The flexible job shop scheduling method based on deep reinforcement learning with transportation time as described in claim 1, characterized in that: In step S3, a policy network and a value network with a linear noise layer are used to improve the network's generalization performance. In step S5, LSTM is added to the policy network and value network of PPO. The superiority of LSTM in processing sequence data is used to fully extract important information from the input state. The information of the input state is increased from one time step to several time steps, making the training process more stable and easier to converge. Combining the idea of ​​transfer networks, a PPO agent with a large network size is first constructed, and it is fully trained using examples of various sizes, so that the policy network and value network can capture information in the workshop state and output excellent actions. Then, a second agent is constructed to solve specific examples, which greatly reduces the computational cost and improves the algorithm's generalization.

5. A flexible job shop scheduling system based on deep reinforcement learning with transportation time, characterized in that: It includes a second intelligent agent, several machines, workpieces, and AGVs, wherein the second intelligent agent applies the flexible workshop scheduling method with transportation time as described in any one of claims 1 to 4 to schedule and control the machines, workpieces, and AGVs.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, invoke all or part of the steps of the method according to any one of claims 1 to 4.

Images