Hybrid flow shop dynamic scheduling optimization method under demand disturbance

Abstract

The invention provides a hybrid flow shop dynamic scheduling optimization method under demand disturbance. Firstly, a Markov decision process (MDP) model is constructed, a state set of the model is composed of a workpiece state and a machine state, and a two-dimensional action set and a specific reward function are designed so as to guide production to be efficiently delivered on time. Secondly, constructing a state feature extraction network of the decision-making agent by adopting a bidirectional long-short term memory network (Bi-LSTM), and processing time sequence dependence; afterwards, in the training process, a near-end strategy optimization algorithm (PPO) is combined with a potential function-based reward shaping method, so that the training stability and efficiency are improved. And finally, in the scheduling process, demand disturbance is continuously monitored, real-time adjustment is performed by using the trained intelligent agent, and smooth and efficient production is ensured. According to the method, the production efficiency is remarkably improved, the resource utilization is optimized, the production flexibility is enhanced, the production cost is reduced, the product quality and customer satisfaction are improved, the intelligent management of the production process is realized, and the market competitiveness and adaptability of enterprises are enhanced.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing and automatic control, and in particular to a method for dynamic scheduling optimization of a hybrid flow shop under demand disturbance. Background Art

[0002] In modern manufacturing, the Hybrid Flow Shop (HFS) is a common form of production organization. It combines the advantages of both assembly line and job shop production modes and has high flexibility and production capacity. However, in the actual production process, due to the rapid changes in market demand and the frequent adjustments of customer orders, the hybrid flow shop often faces the challenge of demand disturbances. These disturbances may include delays in the arrival of workpieces, changes in delivery dates, increases or decreases in the number of workpieces, etc., which directly affect the execution of production plans and the effective allocation of resources. Therefore, how to achieve dynamic scheduling of hybrid flow shops under demand disturbances has become an urgent problem to be solved.

[0003] Traditional methods for the dynamic scheduling problem of hybrid flow shops under demand disturbances mainly include precise calculation, heuristic methods, and meta-heuristic methods. Although the precise calculation method can find the optimal solution, due to the complexity of the problem, its computing time and resource consumption are often too high to meet the needs of real-time scheduling. Heuristic methods rely on empirical rules and simplified assumptions. Although the computational efficiency is high, the quality of the solution is often difficult to guarantee. Meta-heuristic methods, such as genetic algorithms and simulated annealing, improve the quality and computational efficiency of the solution to a certain extent, but still have disadvantages such as low search efficiency and easy to fall into local optimality. Therefore, traditional methods have certain limitations when facing the dynamic scheduling problem of hybrid flow shops under demand disturbances.

[0004] In recent years, with the rapid development of artificial intelligence technology, deep reinforcement learning (DRL), as an emerging machine learning paradigm, has shown great potential in solving complex decision-making problems. Methods based on deep reinforcement learning can automatically adapt to environmental changes by learning the rules and patterns in historical data and output the optimal decision-making strategy. In the dynamic scheduling problem of hybrid flow workshops under demand disturbances, methods based on deep reinforcement learning can perceive changes in the production environment in real time and dynamically adjust production scheduling plans to maximize production efficiency, reduce costs and meet customer needs. In addition, deep reinforcement learning methods also have strong generalization capabilities and can cope with production scheduling problems of different scales and complexities. Therefore, methods based on deep reinforcement learning provide a new idea and approach to solve the dynamic scheduling problem of hybrid flow workshops under demand disturbances. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention proposes a hybrid flow shop robust scheduling method under uncertain demand.

[0006] The technical solution adopted by the present invention to achieve the above-mentioned purpose is:

[0007] A hybrid flow shop dynamic scheduling optimization method under demand disturbance includes the following steps:

[0008] 1) Abstract the dynamic scheduling problem of the hybrid flow shop into a Markov decision process model;

[0009] 2) Construct a decision agent network based on the Markov decision process model;

[0010] 3) Use the proximal policy optimization algorithm to train the decision-making agent network;

[0011] 4) Use the trained decision-making agent network to perform dynamic workshop scheduling.

[0012] The Markov decision process model includes a state set, an action set and a reward function, wherein:

[0013] The state set includes workpiece state and machine state, wherein the workpiece state includes the delivery date of the workpiece, the processing time of the workpiece in each production stage, the stage of the workpiece to be processed and the earliest start time of the stage of the workpiece to be processed; the machine state includes the real-time available time of the machine in each production stage;

[0014] The action set is a two-dimensional space, the first dimension is the selection of the workpiece, that is, which workpiece is selected for processing in the current state; the second dimension is the selection of the machine, that is, which machine is assigned to the selected workpiece for processing;

[0015] The reward function is used to guide the agent to achieve the goal of delivering the workpiece on time.

[0016] The reward function is specifically:

[0017]

[0018] Among them, R is the reward value, W is the positive reward coefficient, D is the workpiece delivery time, T is the actual completion time, P is the negative penalty coefficient, L is the workpiece delay time, and C is the additional cost caused by the delay.

[0019] The decision agent network includes a state feature extraction network and a decision network connected in sequence, wherein:

[0020] The state feature extraction network takes the state sequence as input and outputs a state feature vector;

[0021] The decision network takes state features as vectors as input and outputs action space.

[0022] The state feature extraction network is a Bi-LSTM network.

[0023] The step 3) comprises the following steps:

[0024] 3.1) Parameter initialization: Setting parameters for the proximal strategy optimization algorithm;

[0025] 3.2) Reward shaping calculation: For each time step, calculate the difference between the potential function values ​​of the current state and the next state, and use this difference as the reward after shaping;

[0026] 3.3) Iterative training: loop step 3.2) until the set number of training times is reached, collect the state, action and reward data during the training process, and store them in the buffer;

[0027] 3.4) Policy and value network update: The policy network and value network are updated regularly using the data stored in the buffer.

[0028] The reward R' after shaping is:

[0029] R′=R+V(s′)-V(s)

[0030] V(s)=k·MIT total (s)

[0031] Where R′ is the reward after shaping, R is the original reward obtained at the current state s, V(s) is the potential value of the current state s, V(s') is the potential value of the next state s', and k is a constant used to adjust the impact of idle time on the potential value; MIT total (s) is the sum of the idle times of all machines in the current state s.

[0032] The step 4) comprises the following steps:

[0033] 4.1) Generate initial scheduling plan: Based on the initial state, the initial scheduling plan is generated by adopting the rules of early delivery date priority and earliest idle machine priority;

[0034] 4.2) Real-time monitoring of demand disturbances: Continuously monitor demand disturbances, and immediately proceed to step 4.3) once a specific event occurs, including the arrival of workpieces and changes in delivery dates;

[0035] 4.3) Rescheduling decision: When rescheduling is confirmed, the current state information is input into the decision agent network. The decision agent network gradually schedules actions based on the trained strategy network output, selects the workpieces to be processed and allocates them to the appropriate parallel machines;

[0036] 4.4) Update the scheduling plan: According to the step-by-step scheduling actions output by the decision agent network, the scheduling plan is gradually updated to respond to changes in demand.

[0037] The present invention has the following beneficial effects and advantages:

[0038] 1. Through the real-time decision-making and dynamic adjustment capabilities of the intelligent body, the present invention can quickly and accurately respond to various changes in demand in the production process, thereby greatly improving production efficiency and ensuring a smooth and efficient production process.

[0039] 2. The intelligent agent of the present invention can intelligently allocate machines and human resources according to the production environment and demand disturbances to maximize the utilization of resources, thereby effectively reducing production costs and improving the profitability of the enterprise.

[0040] 3. The present invention also greatly enhances the flexibility of production, making it easy to cope with changes in demand such as the arrival of various workpieces and changes in delivery dates, ensuring that production scheduling can be flexibly adjusted to meet diverse market demands. At the same time, the application of intelligent agents significantly reduces the reliance on manual scheduling, reduces the risk of human error, and improves the accuracy and reliability of production scheduling, thereby further improving product quality and customer satisfaction.

[0041] 4. The present invention deeply integrates artificial intelligence technology into production scheduling, realizing the intelligent management of the production process, which not only improves the operational efficiency of the enterprise, but also enhances the market adaptability and competitiveness of the enterprise, bringing long-term development opportunities and significant economic benefits to the enterprise. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 is a schematic diagram of an action set in the present invention;

[0043] Figure 2 is a flow chart of training an intelligent agent in the present invention;

[0044] Figure 3 is a flow chart of dynamic scheduling in the present invention;

[0045] Figure 4 is a flow chart of the method of the present invention. DETAILED DESCRIPTION

[0046] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0047] The present invention proposes a robust scheduling method for the hybrid flow shop scheduling problem under demand uncertainty. Figure 4 As shown, the following is a detailed description of the specific implementation of the method:

[0048] First, a Markov decision process model is constructed, including a state set, an action set, and a reward function.

[0049] State set: This state set consists of two parts: workpiece state and machine state. The workpiece state includes the delivery date of the workpiece, the processing time of the workpiece in each production stage, the workpiece waiting for processing stage, and the earliest possible start time of the workpiece waiting for processing stage; the machine state includes the real-time available time of the machine in each production stage.

[0050] Action set: The action set is a two-dimensional space. The first dimension is the selection of the workpiece, that is, which workpiece is selected for processing in the current state; the second dimension is the selection of the machine, that is, which machine is assigned to the selected workpiece for processing, such as Figure 1 .

[0051] Reward function: In order to guide the agent to achieve the goal of delivering the workpiece on time, a positive reward is given when the actual completion time of the workpiece does not exceed the delivery date; when the actual completion time exceeds the delivery date, a negative penalty is given according to the delay time and the additional cost incurred. The form of the reward function is:

[0052]

[0053] Where R is the reward value, W is the positive reward coefficient, D is the workpiece delivery time, T is the actual completion time, P is the negative penalty coefficient, L is the workpiece delay time (TD, if T>D), and C is the additional cost caused by the delay.

[0054] Take a specific hybrid flow shop as an example. The shop contains two stages (stage A and stage B), and each stage has two parallel machines (machines A1, A2 and machines B1, B2). There are five workpieces that need to be processed in the shop, and each workpiece has a different processing time in the two stages. The delivery date of the workpiece is also known. During the production process, demand disturbances such as the arrival of new workpieces or changes in delivery dates may be encountered.

[0055] A possible set of states is as follows:

[0056] ●Workpiece status:

[0057] ■Delivery period of each workpiece: D1=5, D2=3, D3=7, D4=4, D5=6

[0058] ■Processing time of each stage: T A1_1 =3,T A1_2 =4,T A1_3 =4,T A1_4 =2,T A1_5 =5,...

[0059] ■The processing stage of each workpiece: Workpiece 1 is waiting for processing in stage 1, workpiece 2 is waiting for processing in stage 2,...

[0060] ■The earliest possible start time of each workpiece to be processed: Workpiece 1 can process phase 1 after time 0, workpiece 2 can process phase 2 after time 3,...

[0061] ●Dynamic status:

[0062] ■The available time of the machines in each production stage: in stage 1, machine 1 is idle after time 5, in stage 1, machine 2 is idle after time 3, ...

[0063] A possible set of actions is as follows:

[0064] ●Action: Workpiece 1, Machine 1 (indicates that in the current state, workpiece 1 is selected and processed on machine 1 in stage 1).

[0065] A possible reward calculation process is as follows:

[0066] ●Assume the following data:

[0067] D=10(delivery period)

[0068] T = 8 (actual completion time)

[0069] W = 2 (positive reward coefficient)

[0070] P = 3 (negative penalty coefficient)

[0071] C = 5 (additional cost caused by delay)

[0072] Because T≤D(8<=10), it is delivered on time. Calculate the reward value: R=+W(DT)=+2(10-8)=+2*2=+4;

[0073] If T = 12 (actual completion time, indicating that the delivery was not on time), then calculate the reward value:

[0074] L=TD=12-10=2, R=-P(L+C)=-3(2+5)=-3*7=-21

[0075] Then, a decision-making agent network is constructed, which consists of two parts: a state feature extraction network and a decision network.

[0076] State feature extraction network: The Bi-LSTM network is used as the state feature extraction network. The Bi-LSTM network can process both the forward and reverse information of the input state sequence, thereby enhancing sensitivity to dynamic changes. In addition, the Bi-LSTM network retains long-term dependency information in the hidden layer, which is essential for capturing long-term trends and cyclical changes in the production process.

[0077] Based on the state feature vector output by the state feature extraction network, a decision network is constructed, which can intelligently select the priority workpieces and the processing machines assigned to them according to the current production status and scheduling goals.

[0078] A possible example of building a decision-making agent network is as follows.

[0079] ●State feature extraction network: set d=39 (indicating the length of the state feature vector), T=20 (indicating that the state sequence contains 20 time steps), and H=128 (indicating that the Bi-LSTM layer has 128 hidden units).

[0080] ●Decision network: Use a fully connected neural network (Dense layer) as the basic architecture, including 2 hidden layers and 1 output layer. Use nonlinear activation functions (such as ReLU) to enhance the network's expressiveness. Set the number of hidden units in the first layer to 256 and the number of hidden units in the second layer to 128.

[0081] Output dimension equals 20

[0082] After that, train the agent. Figure 2 As shown in the figure, the PPO algorithm is used in the process of training the intelligent agent. The PPO algorithm ensures the stability and efficiency of the strategy update by limiting the strategy update amplitude and using multiple old strategy samples for training.

[0083] In order to speed up the training process, a potential function-based reward shaping method is designed. This method uses the prior experience of machine idle time to guide the learning of the agent. By calculating the potential value difference between the current state and the next state, the shaped reward is obtained, thereby encouraging the agent to take actions that can reduce the machine idle time. The calculation formula of the shaped reward R' is as follows:

[0084] R′=R+V(s′)-V(s)

[0085] Among them, R′ is the reward after shaping; R is the original reward obtained in the current state s; V(s) is the potential value of the current state s; V(s') is the potential value of the next state s'. The potential value V(s) is specifically defined as the product of the sum of the idle time of all machines in the current state s and the constant k, that is:

[0086] V(s)=k·MIT total (s)

[0087] where k is a constant used to adjust the effect of idle time on potential value; MIT total (s) is the sum of the idle times of all machines in the current state s.

[0088] A possible example of training an agent is as follows.

[0089] ● Parameter initialization: A series of key parameters need to be set for the PPO algorithm. These parameters include but are not limited to the learning rate (which controls the step size of the network weight update, usually set to a small positive number, such as 0.0003), the discount factor (which determines the current value of future rewards, usually close to 1, such as 0.99, to emphasize the importance of long-term rewards), and the limit of the strategy update range (such as the clipping ratio is set to 0.2 to prevent the strategy update from being too aggressive).

[0090] ●Reward shaping calculation: For each time step, we calculate the difference between the potential function value of the current state and the next state, and use this difference as the reward after shaping. If the action taken by the agent results in a higher potential function value for the next state (i.e., less idle time), it will receive a positive reward; otherwise, it will receive a negative reward.

[0091] ● Iterative training: The training process will be repeated multiple times, each of which contains multiple training cycles (or epochs). In each training cycle, the agent performs actions in the simulated environment according to the current strategy and collects relevant state, action, and reward data. This data is then stored in the experience replay buffer for subsequent network updates.

[0092] ●Strategy and value network update: Use the data stored in the buffer to regularly update the strategy network and value network.

[0093] ● Training effect evaluation: This is achieved by analyzing the performance data of the agent in the environment, such as the on-time delivery rate of workpieces. If the agent can successfully improve the indicators, then the training process can be considered successful.

[0094] Finally, dynamic scheduling. Figure 3As shown in the figure, according to the initial state of the production environment, an initial scheduling plan is generated using the rules of early delivery priority and earliest idle machine priority. During the production process, demand disturbances are continuously monitored, such as the arrival of workpieces and changes in delivery dates. Once these specific events occur, the system will immediately trigger the scheduling adjustment mechanism. At the moment when scheduling adjustment is required, the current production status information is input into the trained decision-making agent. The agent performs step-by-step scheduling actions based on the input status information and the trained policy network output, including selecting the workpieces to be processed and allocating them to appropriate parallel machines. According to the step-by-step scheduling actions output by the agent, the current scheduling plan is gradually updated.

[0095] A possible example of dynamic scheduling is as follows.

[0096] According to the initial state of the production environment, an initial scheduling plan was generated using the rules of "earliest delivery date first" and "earliest idle machine first".

[0097] For example, the initial schedule might be as follows:

[0098] ●Workpiece 2: Stage A (machine A1), Stage B (machine B1)

[0099] ●Workpiece 4: Stage A (machine A2), Stage B (machine B2)

[0100] ●Job 1: Phase A (after machine A1 is idle), Phase B (after machine B1 or B2 is idle)

[0101] ●Job 5: Stage A (after machine A2 is idle), Stage B (after machine B1 or B2 is idle)

[0102] ●Job 3: Phase A (after machine A1 or A2 is idle), Phase B (after machine B1 or B2 is idle)

[0103] During the production process, the demand disturbance is continuously monitored. Assume that at time 3, the following two specific events occur:

[0104] ●Workpiece 6 arrives at the workshop randomly on the second day and needs to be processed immediately.

[0105] ●The delivery date of workpiece 2 is changed to the next day and needs to be completed in advance.

[0106] Once these specific events occur, the system immediately triggers the scheduling adjustment mechanism. At this time, the current production status information needs to be input into the trained decision-making agent.

[0107] The agent gradually outputs scheduling actions based on the input state information and the trained policy network. These actions may include:

[0108] ●Select workpiece 6 for priority processing and assign it to the currently idle machine;

[0109] ● Select workpiece 3 for processing and assign it to machine A2 (assuming machine A2 is idle earlier than machine A1)

[0110] ●Rearrange the processing order and machine allocation of workpiece 1, workpiece 4 and workpiece 5 to ensure that all workpieces can be processed while meeting the delivery date.

[0111] According to the step-by-step scheduling actions output by the agent, the current scheduling plan is gradually updated. The updated scheduling plan may be as follows:

[0112] Artifact 6: Stage A (Machine A1), Stage B (Machine B1)

[0113] Artifact 3: Stage A (Machine A2), Stage B (Machine B2)

[0114] Job 4: Phase A (after machine A1 is idle), Phase B (after machine B1 or B2 is idle, adjusted according to the completion of job 6)

[0115] Job 1: Phase A (after machine A2 is idle), Phase B (after machine B1 or B2 is idle, adjusted according to the completion of job 4)

[0116] Job 5: Phase A (scheduled based on availability of machine A1 or A2), Phase B (scheduled based on availability of machine B1 or B2)

[0117] Job 2: Phase A (scheduled based on availability of machine A1 or A2), Phase B (scheduled based on availability of machine B1 or B2)

[0118] By implementing the method of the present invention, the dynamic scheduling problem of the hybrid flow shop is successfully solved. The method not only improves production efficiency, but also optimizes resource utilization, reduces production costs, and significantly enhances production flexibility.

Claims

1. A method for dynamic scheduling optimization of a hybrid flow shop under demand disturbance, characterized in that: The following steps are involved: 1) Abstract the dynamic scheduling problem of the hybrid flow shop into a Markov decision process model; 2) Construct a decision agent network based on the Markov decision process model; 3) Use the proximal policy optimization algorithm to train the decision-making agent network; 4) Use the trained decision-making agent network to perform dynamic workshop scheduling.

2. The method for dynamic scheduling optimization of a hybrid flow shop under demand disturbance according to claim 1 is characterized in that: The Markov decision process model includes a state set, an action set and a reward function, wherein: The state set includes workpiece state and machine state, wherein the workpiece state includes the delivery date of the workpiece, the processing time of the workpiece in each production stage, the stage of the workpiece to be processed and the earliest start time of the stage of the workpiece to be processed; the machine state includes the real-time available time of the machine in each production stage; The action set is a two-dimensional space, the first dimension is the selection of the workpiece, that is, which workpiece is selected for processing in the current state; the second dimension is the selection of the machine, that is, which machine is assigned to the selected workpiece for processing; The reward function is used to guide the agent to achieve the goal of delivering the workpiece on time.

3. The method for dynamic scheduling optimization of a hybrid flow shop under demand disturbance according to claim 2 is characterized in that: The reward function is specifically: Among them, R is the reward value, W is the positive reward coefficient, D is the workpiece delivery time, T is the actual completion time, P is the negative penalty coefficient, L is the workpiece delay time, and C is the additional cost caused by the delay.

4. The method for dynamic scheduling optimization of a hybrid flow shop under demand disturbance according to claim 1 is characterized in that: The decision agent network includes a state feature extraction network and a decision network connected in sequence, wherein: The state feature extraction network takes the state sequence as input and outputs a state feature vector; The decision network takes state features as vectors as input and outputs action space.

5. The method for dynamic scheduling optimization of a hybrid flow shop under demand disturbance according to claim 4 is characterized in that: The state feature extraction network is a Bi-LSTM network.

6. The method for dynamic scheduling optimization of a hybrid flow shop under demand disturbance according to claim 1 is characterized in that: The step 3) comprises the following steps: 3.1) Parameter initialization: Setting parameters for the proximal strategy optimization algorithm; 3.2) Reward shaping calculation: For each time step, calculate the difference between the potential function values ​​of the current state and the next state, and use this difference as the reward after shaping; 3.3) Iterative training: loop step 3.2) until the set number of training times is reached, collect the state, action and reward data during the training process, and store them in the buffer; 3.4) Policy and value network update: The policy network and value network are updated regularly using the data stored in the buffer.

7. The method for dynamic scheduling optimization of a hybrid flow shop under demand disturbance according to claim 6 is characterized in that: The reward R' after shaping is: R ′ =R+V(s ′ )-V(s) V(s)=k·MIT total (s) Among them, R ′ is the reward after shaping, R is the original reward obtained at the current state s, V(s) is the potential value of the current state s, V(s') is the potential value of the next state s', and k is a constant used to adjust the impact of idle time on the potential value; MIT total (s) is the sum of the idle times of all machines in the current state s.

8. The method for dynamic scheduling optimization of a hybrid flow shop under demand disturbance according to claim 1 is characterized in that: The step 4) comprises the following steps: 4.1) Generate initial scheduling plan: Based on the initial state, the initial scheduling plan is generated by adopting the rules of early delivery date priority and earliest idle machine priority; 4.2) Real-time monitoring of demand disturbances: Continuously monitor demand disturbances, and immediately proceed to step 4.3) once a specific event occurs, including the arrival of workpieces and changes in delivery dates; 4.3) Rescheduling decision: When rescheduling is confirmed, the current state information is input into the decision agent network. The decision agent network gradually schedules actions based on the trained strategy network output, selects the workpieces to be processed and allocates them to the appropriate parallel machines; 4.4) Update the scheduling plan: According to the step-by-step scheduling actions output by the decision agent network, the scheduling plan is gradually updated to respond to changes in demand.

Images