Reentrant hybrid flow shop production scheduling method based on reinforcement learning

Abstract

The invention relates to a reinforcement learning-based production scheduling method for a reentrant hybrid flow shop, and the method comprises the steps: firstly analyzing the production demands of the reentrant hybrid flow shop, and determining the production information of a current product; secondly, determining optimization objectives, taking minimization of the maximum completion time and minimization of the total delay time as the optimization objectives, determining constraint conditions and parameters, determining constraints and related parameter variables in an optimization problem, and establishing a multi-objective reentrant hybrid flow shop scheduling problem model; performing optimization solution by utilizing reinforcement learning, describing and constructing a state space and an action space based on a Markov decision process model, and constructing a global external reward and an internal reward after completion of each stage of production by adopting a layered reward mode in order to overcome challenges caused by a reward sparse problem in reinforcement learning; a bidirectional long short-term memory network (BiLSTM) is used to construct a scheduling strategy, internal scheduling information is extracted, a model is trained by using near-end strategy optimization (PPO), and intelligent decision-making of action selection is realized.

Description

Technical Field

[0001] The invention belongs to the field of production scheduling, and in particular relates to a production scheduling method for a reentrant hybrid flow shop based on reinforcement learning. Background Art

[0002] In the field of manufacturing, the reentrant hybrid flow shop scheduling problem is a complex and critical problem, which involves the reasonable arrangement of production tasks between multiple production lines and multiple processes. This problem is highly dynamic and uncertain because production tasks often have different processing sequences, process requirements and resource constraints. Traditional scheduling methods, such as heuristic algorithms and genetic algorithms, often find it difficult to obtain satisfactory scheduling results when faced with such complex problems. Many existing scheduling methods focus on a single goal, such as minimizing the production cycle or maximizing equipment utilization. However, in actual production, it is often necessary to consider multiple goals at the same time, such as minimizing the maximum completion time and minimizing the total delay time. Single-objective optimization methods often cannot take these goals into account, resulting in unsatisfactory scheduling results. For large-scale, multi-variety, and highly complex production tasks, the computational complexity of traditional scheduling methods is often very high. This not only increases the computational cost and time, but may also affect the accuracy and feasibility of the scheduling results.

[0003] Multi-objective optimization in the reentrant hybrid flow shop scheduling problem refers to the simultaneous consideration of the two objectives of minimizing the maximum completion time and minimizing the total delay time. Minimizing the maximum completion time means shortening the completion time of the latest completed task among all production tasks as much as possible to improve the overall production efficiency. Minimizing the total delay time means minimizing the deviation between the actual completion time and the planned completion time of the production task as much as possible to improve the accuracy and reliability of the production plan. There is a relationship of mutual restraint and balance between these two objectives. For example, in order to minimize the maximum completion time, it may be necessary to sacrifice the priority of some production tasks, resulting in an increase in the total delay time. Conversely, in order to minimize the total delay time, it may be necessary to adjust the allocation and priority of production tasks, thereby affecting the maximum completion time. Therefore, how to balance and optimize between the two objectives is a major challenge in the reentrant hybrid flow shop scheduling problem. In summary, the existing technology has many limitations in solving the multi-objective optimization in the reentrant hybrid flow shop scheduling problem. Summary of the invention

[0004] Aiming at the equipment selection and process sequencing problems in the production of reentrant hybrid flow workshops, a production scheduling method for reentrant hybrid flow workshops based on reinforcement learning was proposed. The scheduling objectives were to minimize the maximum completion time and minimize the total tardiness. Based on the Markov process, a scheduling strategy was constructed using a bidirectional long short-term memory network, and the internal scheduling information was extracted. The proximal policy optimization (PPO) was used to train the model to achieve intelligent decision-making for action selection.

[0005] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0006] A method for production scheduling of a reentrant hybrid flow shop based on reinforcement learning comprises the following steps:

[0007] 1) Based on the relevant information of the production workshop, a mathematical model for the scheduling problem of a reentrant hybrid flow shop is constructed;

[0008] 2) Based on the mathematical model of the reentrant hybrid flow shop scheduling problem, a reinforcement learning model is constructed;

[0009] 3) Based on the bidirectional long short-term memory network, potential scheduling information is extracted, a scheduling strategy model is constructed, the scheduling strategy model is trained using the proximal strategy optimization method, and the trained scheduling strategy model is used to obtain intelligent decisions for action selection.

[0010] The step 1) comprises the following steps:

[0011] 1.1) Determine the processing equipment information and workpiece information;

[0012] 1.2) Taking minimizing the maximum completion time and minimizing the total delay time as the optimization objectives of the reentrant hybrid flow shop production scheduling problem;

[0013] 1.3) Based on the assumptions and actual processing process, determine the constraints and related parameter variables in the re-entrant hybrid flow shop production scheduling problem.

[0014] The constraints include:

[0015]

[0016]

[0017]

[0018]

[0019]

[0020]

[0021] Where i is the workpiece index, I is the workpiece set, I = {1, 2, ..., n}, n is the total number of workpieces, j is the production stage index, J is the production stage set, J = {1, 2, ..., p}, p is the total number of production stages, l is the processing level index, L is the processing level set, L = {1, 2, ..., t}, t is the total number of processing levels, k is the machine index, K j is the set of available machines in production stage j, K j ={1, 2, ...m j},m j is the total number of available machines in production stage j, P ijl is the processing time of job i in production stage j of level 1, S ijl is the start processing time of job i in production stage j at level 1, C ijl is the completion time of workpiece i at production stage j in level l, d i is the delivery time of workpiece i, C max is the maximum completion time, A is a positive number, x ijlk and is the decision variable.

[0022] The step 2) comprises the following steps:

[0023] 2.1) Determine the state space based on the Markov process and the mathematical model of the problem;

[0024] 2.2) Determine the action space;

[0025] 2.3) Design the reward function using a hierarchical reward approach based on the model’s optimization objectives.

[0026] The state space S is:

[0027]

[0028] Among them, W i Represents workpiece i, O ij represents the processing operation of workpiece i at production stage j. Each operation in the job has four characteristics: characteristic 1. the number of layers of the operation, l represents the number of layers the current operation is in; characteristic 2. the scheduling status of the operation, represented by the start processing time S of workpiece i at production stage j at layer l ijl Representation; Feature 3. The processing time of the operation, the processing time P of workpiece i at production stage j in layer l ijl Indication; Features 4. Arrangement of operation equipment.

[0029] The action space A is:

[0030] A=A o ×A m

[0031] Among them, A o represents the operation selection action space, which includes all operations that meet the requirements and those that do not meet the requirements. m It represents the machine selection action space, which also includes all machines that meet the requirements and those that do not meet the requirements. If the workpiece has not been scheduled yet, the operation that meets the requirements is the first operation; if the workpiece has been partially scheduled, the operation that meets the requirements is the next operation after the scheduled operation. For unassigned operations, the machines that meet the requirements are the machines that can be assigned to perform the operation.

[0032] The reward function R is:

[0033]

[0034] Among them, ω1 represents the reward weight of the maximum completion time, and ω2 represents the reward weight of the total delay time.

[0035] In order to achieve the goal of minimizing the maximum completion time and the total delay time, after the agent completes one stage of processing, it will receive two stage rewards R p1 With R p2 :

[0036] R p1 =-(S ijl -C ij-1l )

[0037]

[0038] Among them, C iJl，Max represents the latest processing time of workpiece i in production stage j of layer l. When the actual processing time exceeds C iJl，Max , the workpiece has a risk of delay, then the total stage reward R obtained by the agent after completing a stage of processing p for:

[0039] R p =ω p1 R p1 +ω p2 R p2

[0040] Among them, ω p1 Represents the stage reward R p1 The reward weight, ω p2 Represents the stage reward R p2 The reward weight.

[0041] The step 3) comprises the following steps:

[0042] 3.1) Input the scheduling scenario features into the BiLSTM-based scheduling strategy model;

[0043] 3.2) The scheduling strategy model outputs a feature matrix that combines global information and local information;

[0044] 3.3) Execute the probability distribution of the two sub-problems of network computing operation sorting and machine selection, and select the action with the highest probability among the eligible actions;

[0045] 3.4) Assign the selected jobs to the selected machines and set them at the earliest possible processing position.

[0046] The present invention has the following beneficial effects and advantages:

[0047] 1. Significantly improve production efficiency: By minimizing the maximum completion time, this method can ensure close connection between the various processes on the production line, reduce bottlenecks in the production cycle, and thus significantly improve overall production efficiency. At the same time, minimizing the total delay time helps reduce delays in order delivery, improve customer satisfaction, and further promote the improvement of production efficiency.

[0048] 2. Intelligent decision-making and adaptive capabilities: The use of bidirectional long short-term memory networks (Bi-LSTM) to build scheduling strategies can capture the timing dependencies and internal scheduling information in the production process, providing strong support for intelligent decision-making. The model is trained through the proximal policy optimization (PPO) algorithm, so that the model has the ability to adaptively adjust the scheduling strategy in a complex and changing production environment, thereby coping with various uncertain factors.

[0049] 3. Optimize resource allocation: This method can accurately predict the time and resources required for each process, thereby optimizing resource allocation and reducing resource waste. Through intelligent scheduling, it can ensure the balanced utilization of various equipment, personnel and other resources on the production line and improve resource utilization.

[0050] 4. Promote the intelligent upgrading of manufacturing industry: The successful application of this method will promote the development of manufacturing industry towards intelligence and automation, and enhance the core competitiveness of enterprises. By introducing advanced reinforcement learning technology and intelligent scheduling algorithms, it can bring more efficient and flexible production models to enterprises.

[0051] 5. Reduce production costs: By optimizing production processes and resource allocation, this method helps reduce production costs and improve the profitability of the enterprise. At the same time, reducing production delays and waste also helps reduce the operating costs of the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 Framework diagram of multi-action scheduling strategy based on bidirectional long short-term memory network;

[0053] Figure 2 Bidirectional long short-term memory network operation flow chart. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0055] like Figure 1 and Figure 2 As shown in the figure, consider a reentrant hybrid flow shop scheduling problem. There are N workpieces to be processed, with a total of p processing stages. Each workpiece needs to be processed in t layers, and each layer needs to go through these p stages in sequence, that is, there are t-1 reentrant processes. In each processing stage, there is a processor that processes the specified process, and there is at least one stage with multiple parallel machines. All processors at the same stage can complete the processing of each workpiece, and all parallel machines processing the same process have the same processing rate. The workpiece can enter the next stage only after completing the corresponding process in each stage. When facing parallel processors, the workpiece can only choose one of the parallel machines for processing. The processing time of each process of the workpiece on the corresponding processor is known. Determine the selection of parallel processors for each stage of each workpiece and the arrangement of the processing order of workpieces on the same machine to achieve the goal of minimizing the maximum processing time and minimizing the total delay time.

[0056] Problem Assumptions:

[0057] The process flow of N workpieces is exactly the same, and they are processed t times in sequence through p processing stages.

[0058] The machine is available at any time, regardless of machine failure and maintenance.

[0059] All workpieces arrive at time 0 and can be processed.

[0060] There is no priority between workpieces, that is, once any process of all workpieces is started, it cannot be stopped until the process is completed.

[0061] The transportation time of the workpiece between the workstations at each stage is not taken into account.

[0062] Each machine can only process one workpiece at a time, and each workpiece can only be processed by one machine at any time.

[0063] The present invention provides a method for production scheduling of a reentrant hybrid flow shop based on reinforcement learning, which comprises the following steps:

[0064] S1: Determine the current product production quantity, process requirements, equipment quantity, equipment processability and other information;

[0065] The input of the optimization method includes processing equipment information and workpiece information, and the output is the sorted processing procedure arrangement order, the corresponding maximum completion time and delay time; the processing equipment information includes the type of processing equipment, the number of processing equipment, and the processes that can be processed by the processing equipment; the workpiece information includes the number of workpieces, the workpiece production stage, the number of workpiece production entry and exit layers, the sequence of workpiece processes, and the processing time of the workpiece production stage; the optimization goal of the model is to minimize the maximum completion time of all workpieces and the total delay time.

[0066] S2: Determine the optimization goal, taking minimizing the maximum completion time and minimizing the total delay time as the optimization goal of the reentrant hybrid flow shop production scheduling problem;

[0067] S3: Determine constraints and parameters. Based on the proposed assumptions and the actual processing process, determine the constraints and related parameter variables in the reentrant hybrid flow shop production scheduling problem;

[0068] The parameter symbols of the mathematical model are defined as follows:

[0069] i: artifact index;

[0070] I: Workpiece set, I = {1, 2, ..., n}, n is the total number of workpieces;

[0071] j: production stage index;

[0072] J: set of production stages, J = {1, 2, ..., p}, p is the total number of production stages;

[0073] l: processing level index;

[0074] L: processing level set, L = {1, 2, ..., t}, t is the total number of processing levels;

[0075] k: machine index;

[0076] K j : The set of available machines in production stage j, K j ={1, 2, ...m j},m j is the total number of available machines in production stage j;

[0077] P ijl : The processing time of workpiece i at production stage j in level l;

[0078] S ijl : The start processing time of workpiece i in production stage j at level l;

[0079] C ijl : The completion time of workpiece i at production stage j in level l;

[0080] d i : The delivery date of workpiece i;

[0081] C max : Maximum completion time;

[0082] A: A sufficiently large positive number;

[0083] x ijlk : Decision variable, which takes the value 1 if workpiece i is processed on machine k in production stage j of level l, and 0 otherwise;

[0084] The decision variable takes the value 1 if workpiece i1 is processed before workpiece i2 in production stage j of level l on machine k, and 0 otherwise.

[0085] The mathematical model of the scheduling problem is expressed as follows:

[0086] Optimization goals

[0087] min f1=min C max (1)

[0088]

[0089] Constraints

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] Among them, the optimization goal (1) is to minimize the maximum completion time, and the optimization goal (2) is to minimize the maximum delay time; constraint (3) indicates that the value of the maximum completion time should be greater than or equal to the completion time of the last process of all workpieces at the last level; constraint (4) indicates that each workpiece can only be processed by one machine at each stage of each level; constraint (5) is that any process of all workpieces cannot be stopped once it starts processing until the process is completed; constraints (6) and (7) are process priority constraints, that is, the workpiece can only start the processing of the next stage of the level after completing the previous process in all levels, and can only start the processing of the first process of the next level after completing the last stage of the previous level; constraint (8) ensures that only when a workpiece is processed can the subsequent workpiece be processed; constraint (9) is that the start time of each workpiece in each production stage of each level should not be earlier than time 0; constraint (10) is the value range of the decision variable.

[0099] S4: Determine the state space. Determine the state space in the reinforcement learning process based on the Markov process and the problem mathematical model, including the scheduling state characteristics, processing time characteristics, and processing machine characteristics of each operation;

[0100] Furthermore, the Markov decision process is defined as a four-tuple<S,A,R,P> , where S represents the state space, A represents the action space, R represents the reward function, and P represents the state transition probability matrix.

[0101] Furthermore, the state space S: At each scheduling decision point, the environment features include all job features. Each job feature includes a set of operation sequences, as shown in formula (11),

[0102]

[0103] Where W i Represents workpiece i, O ij represents the processing operation of workpiece i at production stage j. Each operation in the job has four characteristics: characteristic 1. the number of layers of the operation, l represents the number of layers the current operation is in; characteristic 2. the scheduling status of the operation, represented by the start processing time S of workpiece i at production stage j at layer l ijl Represents the processing time of feature 3 operation, which is the processing time P of workpiece i in production stage j of layer l ijl Indication; Features 4. Arrangement of operation equipment.

[0104] S5: Determine the action space. When applying reinforcement learning to solve the scheduling problem, we first need to select the operation to be scheduled and then select the machine to perform the operation. Therefore, a multi-action space design is adopted;

[0105] The action space is represented as: A = A o ×A m , where A o represents the operation selection action space, which includes all operations that meet the requirements and those that do not meet the requirements. m Represents the machine selection action space, which also includes all eligible and non-eligible machines. If the workpiece has not been scheduled yet, its eligible operation is the first operation; if it has been partially scheduled, its eligible operation is the next operation after the scheduled operation. For unassigned operations, its eligible machines are the machines that can be assigned to perform the operation.

[0106] S6: Determine the reward function and design the reward function using a hierarchical reward approach based on the optimization objective of the model;

[0107] Furthermore, the extrinsic reward formulated based on the model’s optimization objective is expressed as: Where ω1 represents the reward weight of the maximum completion time, and ω2 represents the reward weight of the total delay time.

[0108] In order to achieve the goal of minimizing the maximum completion time and the total delay time, after the agent completes one stage of processing, it will receive two stage rewards R p1 With R p2 : R p1 =-(S ijl -C ij-1l ), Among them C iJl，Max represents the latest processing time of workpiece i in production stage j of layer l. When the actual processing time exceeds C iJl，Max When , the workpiece has a risk of delay. Then the total stage reward obtained by the agent after completing a stage of processing is R p =ω p1 R p1 +ω p2 R p2 , where ω p1 Represents the stage reward R p1 The reward weight, ω p2 Represents the stage reward R p2 The reward weight.

[0109] S7: Form an optimization plan, based on the Markov process, use a bidirectional long-short memory network (BiLSTM) to build a scheduling strategy, extract internal scheduling information, and use proximal policy optimization (PPO) to train the model to achieve intelligent decision-making for action selection.

[0110] BiLSTM is used as a bidirectional version of LSTM to extract hidden features in scheduling scenarios. First, the agent observes the scheduling environment and inputs the scheduling scenario features into the BiLSTM-based feature extraction network; secondly, the feature extraction network outputs a feature matrix that combines global information and local information; then, the action network calculates the probability distribution of the two sub-problems of operation sorting and machine selection, and selects the action with the highest probability among the eligible actions; finally, the selected job is assigned to the selected machine and set to the earliest processable position. It is mainly divided into two steps: workpiece feature extraction and environment feature fusion. In the job feature extraction step, the state features of all operation sequences of workpiece i are input into the BiLSTM network to obtain the positive feature h f and the reverse feature h r Then these two features are connected to form the state feature of workpiece i (h f ,h r ). Since the network input structure from different artifacts is consistent and the gating parameters within the network are consistent, the extracted encoding is uniform. In the environmental feature fusion step, the average value of all job features is calculated to represent the environmental state feature H.

[0111] In summary, the reentrant hybrid flow shop production scheduling method based on reinforcement learning has significant beneficial effects. It can not only improve production efficiency and optimize resource allocation, but also improve production flexibility and response speed, reduce production costs, and promote the intelligent upgrading of manufacturing industry. These advantages will bring significant economic and social benefits to enterprises.

Claims

1. A reentrant hybrid flow shop production scheduling method based on reinforcement learning, characterized in that: The following steps are involved: 1) Based on the relevant information of the production workshop, a mathematical model for the scheduling problem of a reentrant hybrid flow shop is constructed; 2) Based on the mathematical model of the reentrant hybrid flow shop scheduling problem, a reinforcement learning model is constructed; 3) Based on the bidirectional long short-term memory network, potential scheduling information is extracted, a scheduling strategy model is constructed, the scheduling strategy model is trained using the proximal strategy optimization method, and the trained scheduling strategy model is used to obtain intelligent decisions for action selection.

2. A method for production scheduling of a reentrant hybrid flow shop based on reinforcement learning according to claim 1, characterized in that: The step 1) comprises the following steps: 1.1) Determine the processing equipment information and workpiece information; 1.2) Taking minimizing the maximum completion time and minimizing the total delay time as the optimization objectives of the reentrant hybrid flow shop production scheduling problem; 1.3) Based on the assumptions and actual processing process, determine the constraints and related parameter variables in the re-entrant hybrid flow shop production scheduling problem.

3. A method for production scheduling of a reentrant hybrid flow shop based on reinforcement learning according to claim 2, characterized in that: The constraints include: Where i is the workpiece index, I is the workpiece set, I = {1, 2, ..., n}, n is the total number of workpieces, j is the production stage index, J is the production stage set, J = {1, 2, ..., p}, p is the total number of production stages, l is the processing level index, L is the processing level set, L = {1, 2, ..., t}, t is the total number of processing levels, k is the machine index, K j is the set of available machines in production stage j, K j ={1,2,...m j },m j is the total number of available machines in production stage j, P ijl is the processing time of workpiece i at production stage j in layer l, S ijl is the start processing time of workpiece i in production stage j at level l, C ijl is the completion time of workpiece i at production stage j in level l, d i is the delivery time of workpiece i, C max is the maximum completion time, A is a positive number, x ijlk and is the decision variable.

4. The method for production scheduling of a reentrant hybrid flow shop based on reinforcement learning according to claim 1, characterized in that: The step 2) comprises the following steps: 2.1) Determine the state space based on the Markov process and the mathematical model of the problem; 2.2) Determine the action space; 2.3) Design the reward function using a hierarchical reward approach based on the model’s optimization objectives.

5. A method for production scheduling of a reentrant hybrid flow shop based on reinforcement learning according to claim 4, characterized in that: The state space S is: Among them, W i Represents workpiece i, O ij represents the processing operation of workpiece i at production stage j. Each operation in the job has four characteristics: characteristic 1. the number of layers of the operation, l represents the number of layers the current operation is in; characteristic 2. the scheduling status of the operation, represented by the start processing time S of workpiece i at production stage j at layer l ijl Representation; Feature 3. The processing time of the operation, the processing time P of workpiece i at production stage j in layer l ijl Indication; Features 4. Arrangement of operation equipment.

6. A method for production scheduling of a reentrant hybrid flow shop based on reinforcement learning according to claim 4, characterized in that: The action space A is: A=A o ×A m Among them, A o represents the operation selection action space, which includes all operations that meet the requirements and those that do not meet the requirements. m It represents the machine selection action space, which also includes all machines that meet the requirements and those that do not meet the requirements. If the workpiece has not been scheduled yet, the operation that meets the requirements is the first operation; if the workpiece has been partially scheduled, the operation that meets the requirements is the next operation after the scheduled operation. For unassigned operations, the machines that meet the requirements are the machines that can be assigned to perform the operation.

7. A method for production scheduling of a reentrant hybrid flow shop based on reinforcement learning according to claim 4, characterized in that: The reward function R is: Among them, ω1 represents the reward weight of the maximum completion time, and ω2 represents the reward weight of the total delay time.

8. The method for production scheduling of a reentrant hybrid flow shop based on reinforcement learning according to claim 7 is characterized in that: In order to achieve the goal of minimizing the maximum completion time and the total delay time, after the agent completes one stage of processing, it will receive two stage rewards R p1 With R p2 : R p1 =-(S ijl -C ij-1l ) Among them, C iJl,Max represents the latest processing time of workpiece i in production stage j of layer l. When the actual processing time exceeds C iJl,Max , the workpiece has a risk of delay, then the total stage reward R obtained by the agent after completing a stage of processing p for: R p =ω p1 R p1 +oh p2 R p2 Among them, ω p1 Represents the stage reward R p1 The reward weight, ω p2 Represents the stage reward R p2 The reward weight.

9. The method for production scheduling of a reentrant hybrid flow shop based on reinforcement learning according to claim 1, characterized in that: The step 3) comprises the following steps: 3.1) Input the scheduling scenario features into the BiLSTM-based scheduling strategy model; 3.2) The scheduling strategy model outputs a feature matrix that combines global information and local information; 3.3) Execute the probability distribution of the two sub-problems of network computing operation sorting and machine selection, and select the action with the highest probability among the eligible actions; 3.4) Assign the selected jobs to the selected machines and set them at the earliest possible processing position.

Images