A deep reinforcement learning driven distributed heterogeneous zero-wait flow shop scheduling optimization system

Abstract

The present application relates to the technical field of distributed green production scheduling, and particularly relates to a deep reinforcement learning driven distributed heterogeneous zero-wait flow shop scheduling optimization system for minimizing total tardiness time and total energy consumption; comprising the following steps: a workpiece sequence initialization module, an optimal allocation rule based on a random sequence is designed to generate a better initial solution; an optimization module based on a migration bird algorithm, an improved migration bird optimization algorithm framework is designed to optimize the performance of the solution; a deep reinforcement learning driven decision module, the result of deep reinforcement learning training is embedded in the migration bird algorithm optimization module for operator selection; a knowledge driven energy saving module, according to the characteristics of the distributed heterogeneous zero-wait flow shop scheduling problem under energy constraints, a rule for extending the workpiece processing time on a specific machine is designed to reduce the total tardiness time of the workpiece and the processing energy consumption.

Description

Technical Field

[0001] This invention relates to the field of distributed green production scheduling technology, specifically to a deep reinforcement learning-driven distributed heterogeneous zero-wait flow shop scheduling optimization system (MHA-PG). Background Technology

[0002] With globalization, distributed manufacturing has become an unstoppable trend due to its better fault tolerance, scalability, and processing speed. The shop floor scheduling problem refers to optimizing time, cost, and energy consumption by processing workpieces in a reasonable order using limited resources and equipment. Due to the large number of factories, workpieces, and machines, the flow shop scheduling problem is considered an NP-hard problem with a huge and complex solution space. Distributed shop floor scheduling involves assigning workpieces to different factories for simultaneous processing according to certain rules. Compared to the single-factory shop floor scheduling problem, distributed shop floor scheduling considers coordination and communication mechanisms between multiple factories. As the number of factories, workpieces, and machines increases, the complexity of the scheduling problem also increases significantly. Therefore, developing effective optimization systems is of great significance in production scheduling research and practical applications.

[0003] The scheduling problem of distributed heterogeneous zero-wait flow shop under energy constraints is a crucial issue in distributed manufacturing and green manufacturing. Its core lies in employing appropriate optimization methods to obtain effective solutions. Traditional mathematical methods, such as branch and bound and linear programming, can yield the most accurate solutions, but due to the need to construct complex models tailored to specific problems, they are often only applicable to small-scale shop scheduling problems. For large-scale practical production applications, metaheuristic algorithms are widely used due to their simple structure and strong applicability. The Migrating Birds Optimization Algorithm (MBO), as a metaheuristic algorithm, balances exploration and development capabilities well with its unique information gain mechanism and leader replacement mechanism, thus being widely used to solve shop scheduling problems. Deep reinforcement learning is a powerful decision optimization method with the ability to autonomously mine problem features and accumulate problem information for decision optimization, and can be used to guide algorithms to quickly find effective feasible solutions within the solution space. This invention utilizes the training results of deep reinforcement learning to guide the Migrating Birds Optimization Algorithm, thereby obtaining high-quality feasible solutions. Summary of the Invention

[0004] This invention provides a deep reinforcement learning-driven distributed heterogeneous zero-wait flow shop scheduling optimization system, which is used to optimize the maximum completion time and total energy consumption of the distributed heterogeneous zero-wait flow shop scheduling problem under energy consumption constraints.

[0005] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0006] A deep reinforcement learning-driven distributed heterogeneous zero-wait flow shop scheduling optimization system includes the following steps:

[0007] Step 1: Workpiece sequence initialization module. In this module, an optimal allocation rule based on a random sequence is designed to initialize the population.

[0008] Step 2: Optimization module based on migratory bird algorithm. In this module, a framework based on migratory bird optimization algorithm is designed to further optimize feasible solutions;

[0009] Step 3: Deep reinforcement learning-driven decision-making module. In this module, a decision-making mechanism based on deep reinforcement learning is designed to fit the selection probabilities of different operation operators. The data generated during the iteration process is abstracted into knowledge to guide the training of deep neural networks. The training results of deep reinforcement learning are used to guide the selection of operation operators in the migratory bird algorithm optimization module.

[0010] Step 4: Knowledge-driven energy-saving module. In this module, based on the characteristics of the distributed heterogeneous zero-wait flow shop scheduling problem under energy consumption constraints, a rule is designed to simultaneously optimize two objectives, reducing the total delay time of workpieces while reducing processing energy consumption.

[0011] Preferably, in step 1, the optimal allocation rule based on the random sequence first generates a certain number of random sequences, with each individual corresponding to a random sequence; secondly, the workpieces to be processed are tentatively inserted into different positions in the factory in the order of the sequence, and the final position is the position with the shortest completion time; finally, the solutions corresponding to each individual are constructed into a concurrent population.

[0012] Preferably, in step 2, firstly, all individuals in the population are arranged into a V-shaped topology according to their quality, with the individual at the front of the topology being regarded as the leader bird and the remaining individuals as followers. Then, a neighborhood search is performed on each individual according to eight different operation operators, where the optimal solution generated by the neighborhood search is used to evolve itself, and the second-best solution is shared with the subsequent followers as its neighborhood solution. Finally, when the algorithm iterates to a certain stage, the leader bird moves to the tail of the topology, and the subsequent followers fill the position to become the leader bird.

[0013] Preferably, in step 3, a deep neural network is constructed in this module. The fitting result of the deep neural network is used as the selection probability of the operation operator, and the policy gradient method is used as the optimizer of the deep neural network to optimize the parameters of the neural network based on the gradient information of the generated data.

[0014] Preferably, in step 4, two workpieces processed consecutively on the same machine are considered as a node. If the current workpiece and the previous workpiece have a node on the subsequent processing machine, and the current workpiece and the next workpiece have a node on the preceding processing machine, the workpiece processing time on the machine between the two nodes is extended, thereby reducing the total delay time of the workpiece and reducing processing energy consumption.

[0015] The present invention has the following beneficial effects:

[0016] (1) This invention analyzes the characteristics of the distributed heterogeneous zero-wait flow shop scheduling problem under energy consumption constraints, constructs a hybrid linear integer programming model for this type of problem, and proposes an evaluation criterion that simultaneously considers total delay time and total energy consumption.

[0017] (2) This invention proposes a deep reinforcement learning-driven migratory bird optimization algorithm to solve this problem. Eight neighborhood operation operators are designed, and a deep reinforcement learning method is used to achieve fully automated adjustment of the selection probability of the operation operators.

[0018] (3) Based on the characteristics of the problem, the present invention designs a knowledge-driven energy-saving strategy that can simultaneously optimize total delay time and total energy consumption without changing the workpiece process arrangement.

[0019] (4) The present invention is simple in logic, easy to implement and extend. This optimization system can be extended to most scheduling problems in manufacturing production systems, such as location problems and vehicle routing problems. Attached Figure Description

[0020] Figure 1 This is a schematic diagram of the structure of the distributed heterogeneous zero-wait flow workshop scheduling optimization system under energy consumption constraints provided by the present invention.

[0021] Figure 2 This is an example Gantt chart of the problem in this invention.

[0022] Figure 3 This is a schematic diagram of the migratory bird optimization algorithm framework in this invention.

[0023] Figure 4 This is a schematic diagram of a deep network based on policy gradient in this invention.

[0024] Figure 5 This is a graph showing the experimental results comparing our system with other systems in solving this problem.

[0025] Figure 6 This system is used to solve Pareto front plots in conjunction with other systems. Detailed Implementation

[0026] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0027] Example 1

[0028] A deep reinforcement learning-driven distributed heterogeneous zero-wait flow shop scheduling optimization system, such as Figure 1 As shown, it includes the following steps:

[0029] Step 1: Workpiece Sequence Initialization Module. In this module, an optimal allocation rule based on random sequences is designed to generate better initial solutions. First, a certain number of random sequences are generated, with each individual corresponding to one random sequence. Second, the workpieces to be processed are tentatively inserted into different positions in the factory according to the sequence order, with the final position being the one with the shortest completion time. Finally, the solutions corresponding to each individual are constructed into a concurrent population. The pseudocode of the optimal allocation rule based on random sequences is shown in Algorithm 1.

[0030]

[0031] Step 2: Optimization Module Based on Migratory Bird Algorithm. This module designs a framework based on the migratory bird optimization algorithm to optimize solution performance. First, all individuals in the population are arranged into a V-shaped topology according to their quality, with the best individual at the front of the topology being the leader bird and the remaining individuals being followers. Then, a certain number of neighborhood searches are performed on each individual, using operators including velocity-based insertion operator (V-Swap), velocity matrix shuffling operator (V-Shuffle), and sequence-based exchange operator (S-Shuffle). The algorithm employs eight operation operators: Swap, S-Insert, C-InnerInvert, C-InnerSwap, C-InterInvert, and C-InterSwap. Furthermore, the best solution generated by an individual's neighborhood search is used to evolve itself, while the second-best solution is shared with subsequent birds as its neighborhood solution. Finally, at a certain stage of evolution, the leader bird moves to the tail of the topology, and the following birds fill the gap to become the leader bird. The pseudocode based on the migratory bird optimization algorithm framework is shown in Algorithm 2.

[0032]

[0033] Step 3: Deep Reinforcement Learning-Driven Decision Module. In this module, a deep reinforcement learning-driven neighborhood search mechanism is designed. The results of deep reinforcement learning training are used for operator selection in the migratory bird algorithm optimization module. This module constructs a deep neural network and uses the policy gradient method as the optimizer of the deep neural network. The output of the deep network is the selection probability of the operator. The deep reinforcement learning-driven decision module can continuously learn and adjust the selection probability of the operator during the algorithm iteration process to reduce invalid searches. The pseudocode of the deep reinforcement learning-driven neighborhood search mechanism is shown in Algorithm 3.

[0034]

[0035] The policy gradient-based neighborhood search mechanism includes eight different operators for updating the neighborhood solution. V-Swap and V-Shuffle are based on the velocity matrix, S-Swap and S-Insert are based on processing steps, and C-InnerSwap, C-InterSwap, C-InnerInsert, and C-InterInvert are based on the critical factory. In a distributed scenario, the final processing time of a workpiece is determined by the factory with the longest processing time; therefore, this factory is called the critical factory, and the other factories are called non-critical factories. The eight operators are shown below:

[0036] V-Swap: From the velocity matrix Two velocity vectors are randomly selected from the data. and And exchange their positions.

[0037] V-Shuffle: Random selection and Two positions, shuffled middle and All velocity vectors between.

[0038] S-Swap: From the process Two workpieces are randomly selected from the middle. and Exchange of assignments and The location.

[0039] S-Insert: From Randomly select a workpiece , to the workpiece Insert into a random position in the process middle.

[0040] C-InnerSwap: Randomly selects two workpieces from the critical plant. and ,exchange and The location.

[0041] C-InterSwap: Randomly selects a workpiece from a key factory. , to the workpiece Random workpieces in non-critical factories Exchange positions.

[0042] C-InnerInsert: Randomly selects an item from the critical plant. , to the workpiece Insert into a random location in the critical factory. middle.

[0043] C-InterInsert: Randomly selects a workpiece from a critical plant. , to the workpiece Insert into a random location in a non-critical factory. middle.

[0044] Step 4: Knowledge-driven energy-saving module. In this module, based on the characteristics of the distributed heterogeneous zero-wait flow shop scheduling problem under energy consumption constraints, a rule is designed to extend the processing time of workpieces on specific machines, thereby reducing the total delay of workpieces and reducing processing energy consumption.

[0045] Preferably, in step 4, two workpieces processed consecutively on the same machine are considered as a node. If the current workpiece and the previous workpiece have a node on the subsequent processing machine, and the current workpiece and the next workpiece have a node on the preceding processing machine, the workpiece processing time on the machine between the two nodes is extended, thereby reducing the total delay of the workpiece and reducing processing energy consumption.

[0046] Example 2

[0047] like Figure 1 As shown, Figure 1 This is a schematic diagram of the optimized system structure for the energy-constrained distributed heterogeneous zero-wait flow shop scheduling problem provided by the present invention. The system includes a workpiece sequence initialization module, an optimization module based on the migratory bird algorithm, a deep reinforcement learning-driven decision-making module, and a knowledge-driven energy-saving module.

[0048] like Figure 2 As shown, Figure 2 This is a Gantt chart illustrating an example of the problem solved in this invention. For the distributed heterogeneous zero-wait flow shop scheduling problem under energy constraints, Each workpiece is assigned to Each factory processes [the products], and each factory has [a certain number of factories]. The process is in Processing is done on machines. The processing flow within each factory is identical, satisfying zero-wait constraints. Each machine in the factory has a processing speed. The speed selected for each operation cannot be changed before the operation is completed. As shown in the diagram, 8 workpieces are assigned to 2 factories, each with 3 processing machines, and there is no waiting time for each workpiece during its processing on different machines.

[0049] like Figure 3 As shown, Figure 3 This is a schematic diagram of the migratory bird optimization algorithm framework. A population of N individuals is generated and arranged in a V-shaped topology according to their target value. The best individual at the front of the topology is the leader bird, and the remaining individuals are followers. Each individual undergoes a certain number of neighborhood searches using eight operators: velocity-based insertion, velocity matrix disorder, sequence-based exchange and insertion, key factory insertion, key factory exchange, key factory and non-key factory insertion, and key factory and non-key factory exchange. The best solution generated by an individual's neighborhood search is used to evolve the individual itself, while the second-best solution is shared with subsequent followers as its neighborhood solution. The leader bird guides the entire population to accelerate evolution through this sharing mechanism. At a certain stage of evolution, the leader bird moves to the tail of the topology, and a follower bird takes its place as the leader. This leader bird replacement mechanism effectively enhances population diversity and avoids premature convergence of the algorithm.

[0050] like Figure 4 As shown, Figure 4 This is a schematic diagram of a deep network based on policy gradient, constructing the iterative process of the migratory bird optimization algorithm as a Markov decision process. The trajectory network information of all individuals in the population is considered as the input layer of the deep neural network. As the algorithm iterates, the input information continuously changes, passing through multiple hidden layers and undergoing softmax normalization, ultimately outputting the selection probabilities of different operators. The resulting information guides the input of the next generation of trajectories, thus guiding the algorithm's progression. The update formula for the policy gradient is as follows:

[0051]

[0052] like Figure 5 As shown, Figure 5This is a comparison chart of our system and other systems in solving this problem. To analyze the effectiveness of our system in solving this problem, we have compared it with several other systems. The selected comparison system is: KMOEA (J.-Q. Li, 2022.), CMA (J.-j. Wang, and L. Wang, "A Cooperative MemeticAlgorithm with Learning-Based Agent for Energy-Aware Distributed Hybrid Flow-Shop Scheduling," IEEE Transactions on Evolutionary Computation, pp. 1-1, 2021.), MOWSA (G. Wang, L. Gao, X. Li et al., "Energy-efficient distributedpermutation flow shop scheduling problem using a multi-objective whale swarmalgorithm," Swarm and Evolutionary Computation, vol. 57, 2020.)

[0053] Figure 5The test results of the system and the comparison system on hypervolume (HV) and CMetric under different scale instances are presented respectively. The comparison results show that the system outperforms the latest and best comparison optimization system under different scale instances. The Friedman test shows that the system and the comparison system have statistically significant differences. There are two reasons for this: (1) The migratory bird optimization algorithm driven by deep reinforcement learning adjusts the selection probability of different operation operators according to the results of each generation during the iteration process, reducing meaningless searches and thus obtaining a better quality solution; (2) The optimization system summarizes the knowledge of the distributed heterogeneous zero-wait flow shop scheduling problem under energy consumption constraints, designs a knowledge-driven energy-saving strategy, and obtains a higher quality feasible solution.

[0054] like Figure 6 As shown, Figure 6 This is a Pareto front plot of our system and other systems under conditions where the number of plants ranges from 2 to 7. By comparison, the Pareto front obtained by MHA-PG is closer to the origin, which means that the convergence accuracy of the non-dominated solutions obtained by MHA-PG is better than that of KMOEA, CMA, and MOWSA. Furthermore, the approximate distribution of the non-dominated solutions obtained by MHA-PG is more uniform and compact, indicating that the diversity of the Pareto front is better than that of the comparison systems. Figure 6 The results show that MHA-PG is an effective system for solving the distributed heterogeneous zero-wait flow shop scheduling problem under energy consumption constraints, and its Pareto solution set is closer to the Pareto optimal solution.

[0055] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A deep reinforcement learning-driven distributed heterogeneous zero-wait flow shop scheduling optimization system, characterized in that, Includes the following steps: Step 1: Workpiece Sequence Initialization Module. In this module, an optimal allocation rule based on random sequences is designed to generate a better initial solution. First, the optimal allocation rule based on random sequences generates a certain number of random sequences, with each individual corresponding to one random sequence. Second, the workpieces to be processed are tentatively inserted into different positions in the factory according to the sequence order, with the final position being the one with the shortest completion time. Finally, the solutions corresponding to each individual are constructed into a concurrent population. Step 2: Optimization Module Based on Migratory Bird Algorithm. In this module, a framework based on the migratory bird optimization algorithm is designed to optimize the performance of the solution. First, all individuals in the population are arranged into a V-shaped topology according to their quality. The individual at the front of the topology is considered the leader bird, and the remaining individuals are considered followers. Then, a certain number of neighborhood searches are performed on each individual. The search operators include eight types of operations: velocity-based insertion operator, velocity matrix disorder operator, sequence-based exchange operator and insertion operator, key factory insertion operator, key factory exchange operator, key factory and non-key factory insertion operator, and key factory and non-key factory exchange operator. Furthermore, the best solution generated by the neighborhood search of an individual is used to evolve itself, and the second-best solution is shared with subsequent followers as its neighborhood solution. Finally, after evolving to a certain stage, the leader bird moves to the tail of the topology, and the followers take over to become the leader bird. Step 3: Deep reinforcement learning-driven decision module. In this module, the results of deep reinforcement learning training are used for operator selection in the migratory bird algorithm optimization module. This module constructs a deep neural network and uses the policy gradient method as the optimizer of the deep neural network. The output of the deep network is the selection probability of the operator. The deep reinforcement learning-driven decision module can continuously adjust the selection probability of the operator during the algorithm iteration process to reduce invalid search. Step 4: Knowledge-driven energy-saving module. In this module, two workpieces processed consecutively on the same machine are regarded as a node. If the current workpiece and the previous workpiece have a node on the subsequent processing machine, and the current workpiece and the next workpiece have a node on the preceding processing machine, the workpiece processing time on the machine between the two nodes is extended. This reduces the total delay of the workpiece and reduces processing energy consumption.

Images