Dynamic hybrid flow shop scheduling method considering deterioration effect

Abstract

The invention discloses a dynamic hybrid flow shop scheduling method considering a deterioration effect, and the method comprises the steps: building an environment framework of a hybrid flow shop, initializing the environment parameters, setting the constraint conditions of shop scheduling, and enabling the scheduling target to be the minimization of the total cost; defining a state space, an action space and a reward mechanism; a deterioration function is designed, and the machining time and cost of the machine under different deterioration degrees are calculated; setting a forced maintenance rule; related information of the workpiece and the machine is updated; running a DQN algorithm for learning and training; recording a scheduling result, judging whether a training round reaches a termination condition or not, if so, performing the next step, otherwise, returning to the previous step; and drawing a total cost iterative graph, the longitudinal axis being the total cost after each scheduling is completed, the horizontal axis being the scheduling completion number, and obtaining a workshop scheduling result with the lowest cost. According to the method, the DQN algorithm is used for solving the dynamic hybrid flow shop scheduling problem considering the deterioration effect, and a new scheduling rule is designed, so that the scheduling is more efficient.

Description

Technical Field

[0001] The invention belongs to the field of production and scheduling, and in particular relates to a dynamic hybrid flow shop scheduling method taking deterioration effects into consideration. Background Art

[0002] Workshop production is a major production method in the manufacturing industry. With the development of society, the single and fixed production mode of traditional manufacturing is no longer applicable. Workshop production also needs to be intelligent, and it is necessary to consider more complex production environments and use more advanced workshop scheduling methods to cope with more diverse production needs. Researching and solving workshop scheduling problems can help manufacturing companies reduce costs and increase efficiency, shorten production time and improve production quality while minimizing the impact of disturbances, thereby helping companies reduce production costs and improve their competitiveness.

[0003] The Hybrid Flow Shop Scheduling Problem (HFSP) is a common workshop scheduling problem. It is based on the Flow Shop Scheduling Problem (FSP) and introduces multiple selectable parallel machines in all or part of the processing stages. This method of introducing parallel machines makes production more flexible, improves the flexibility and fault tolerance of production, can respond quickly when facing emergencies, and has strong anti-interference ability. In actual production, the state of the machine will change with the processing time, resulting in a deterioration effect. The longer the processing time, the worse the state of the machine, which may affect the machine processing time and increase unnecessary processing costs. Therefore, it is also very important to consider the deterioration effect of the machine in the hybrid flow shop scheduling problem and maintain the machine at a certain frequency. In recent years, many effective algorithms have been applied to solve the hybrid flow shop scheduling problem, such as exact algorithms, heuristic algorithms and meta-heuristic algorithms, among which the ability of meta-heuristic algorithms to solve problems is more significant.

[0004] The Deep Reinforcement Learning (DRL) algorithm is a meta-heuristic algorithm that combines deep learning and reinforcement learning. It can be trained and learned in a black box, and can make intelligent decisions by learning the interaction between the agent and the environment. It has advantages in solving combinatorial optimization problems such as workshop scheduling and large-scale problems. However, in existing research, deep reinforcement learning algorithms have not been used to solve the hybrid flow workshop scheduling problem while considering the impact of machine deterioration on scheduling. Summary of the invention

[0005] Based on the above problems, the present invention provides a dynamic hybrid flow shop scheduling method considering the deterioration effect, which is based on the DQN algorithm and takes minimizing the total cost as the goal. By setting the deterioration effect function and maintenance cost, the workpiece scheduling is optimized.

[0006] The technical solution adopted by the present invention is as follows:

[0007] Step 1: Build the environment framework of the hybrid flow shop and initialize the environment parameters, set the constraints of the shop scheduling, and set the scheduling goal to minimize the total cost;

[0008] Step 2: Define the state space, action space and reward mechanism;

[0009] Step 3: Design a deterioration function to calculate the processing time and cost of the machine under different deterioration levels;

[0010] Step 4: Set mandatory maintenance rules. When the machine deteriorates to a certain extent, mandatory maintenance will be performed.

[0011] Step 5: Update the relevant information of the workpiece and machine;

[0012] Step 6: Run the DQN algorithm for learning and training;

[0013] Step 7: Record the scheduling results and determine whether the training round has reached the termination condition. If so, execute step 8, otherwise return to step 6;

[0014] Step 8: Draw a total cost iteration graph, with the vertical axis representing the total cost after each scheduling completion and the horizontal axis representing the number of scheduling completions. Analyze the scheduling results based on the iteration graph to obtain the workshop scheduling result with the lowest cost.

[0015] Furthermore, in step 1, before setting up the environment, the scheduling problem and scheduling goals must be clarified. The present invention considers a mixed assembly line workshop with different parallel machines, in which there are n workpieces and m machines, each workpiece has the same p processes, each process has a machine set, which is composed of machines that can process the process, and in the same machine set, the processing time of the same process of the same workpiece on each machine is not exactly the same. At the same time, the workshop also has some constraints: each workpiece is processed according to a fixed process sequence, and each process can only be processed on a machine in the specified machine set; each machine can only process one process at the same time; each process is only processed once, and the next process can only be carried out after the previous process is completed; each process cannot be interrupted during the processing; except for processing, maintenance and waiting time, other time consumption is not considered. The environmental parameters and mathematical symbols involved in the present invention are as follows:

[0016] n: number of workpieces;

[0017] m: number of machines;

[0018] p: number of workpiece processes;

[0019] r: number of remaining processes;

[0020] J i : the i-th workpiece;

[0021] O ij : The jth process of the i-th workpiece;

[0022] M k : the kth machine;

[0023] M ij :Can process process O ij A collection of machines;

[0024] T ijk :Process O ij On machine M k Processing time on

[0025] Process O ij In M ij Average processing time in

[0026] AT ijk :After the machine deteriorates, process O ij On machine M k The actual processing time on

[0027] CT i : The time when the most recent process of workpiece i is completed;

[0028] CT k : The time when the machine last completed processing;

[0029] S i : The process that workpiece i will need to be processed;

[0030] O r : The number of unfinished processing steps for all workpieces;

[0031] UN i : A collection of unfinished workpieces;

[0032] T k : Machine processing time;

[0033] A k : Machine age (after maintenance operations, the machine age will be restored to a certain extent);

[0034] A ave : Average age of all machines;

[0035] U k : Machine utilization;

[0036] U ave : The average utilization of all machines;

[0037] d: deterioration factor;

[0038] D a : The age of the machine at the turning point of machine deterioration, a=1, 2 or 3; where D1, D2 and D3 represent the maximum age of the machine in the first, second and third deterioration stages respectively. When the machine age reaches and exceeds the corresponding deterioration turning point, the machine enters a new deterioration stage;

[0039] D k : Machine deterioration coefficient;

[0040] mt a : The number of maintenance times for each type of maintenance, a=1,2 or 3; where mt1, mt2 and mt3 represent the number of maintenance times for minor repairs, medium repairs and major repairs respectively;

[0041] CM a : The cost of each type of maintenance, a = 1, 2 or 3; CM1, CM2 and CM3 represent the cost of a minor repair, a medium repair and a major repair respectively;

[0042] TM a : The time required for each type of maintenance, a = 1, 2 or 3; TM1, TM2 and TM3 represent the time required for a minor repair, a medium repair and a major repair respectively;

[0043] mq: the number of times the mandatory maintenance is performed;

[0044] C t : Machine operation cost per unit time;

[0045] C s : The unit cost after each processing step or each maintenance completed;

[0046] C: total cost;

[0047] T: system running time;

[0048]

[0049] The goal of the present invention is to minimize the total cost, which can be specifically expressed by the formula:

[0050]

[0051] The C TIt represents the additional repair cost of mandatory maintenance on the machine. The mandatory maintenance cost of the machine = the cost required for an overhaul + the additional repair cost.

[0052] Furthermore, in step 2, the state space, action space, and reward mechanism are crucial in the DQN algorithm. The algorithm learns, explores, and trains by observing the initial state, then selecting an action from the action space at each time step and observing the next state and reward returned by the environment to form experience.

[0053] The state space of the present invention includes five state values, namely, the average service life of all machines, the single-round cost, the remaining process, the average utilization rate of the machine, and the machine deterioration coefficient:

[0054] (6) Average age of all machines:

[0055]

[0056] A k (t) represents the machine age at the current time t;

[0057] (7) Cost per round:

[0058]

[0059] (8) Remaining processes: After each process is completed, the number of remaining processes r decreases by 1.

[0060] (9) Average machine utilization rate:

[0061]

[0062] U k (t) represents the machine utilization at the current time t;

[0063] (10) Machine deterioration coefficient:

[0064]

[0065] The action space of the present invention includes ten actions, including 7 processing actions and 3 maintenance actions. The 7 processing actions are the shortest processing time rule of the process, the longest processing time rule of the process, SPT, LPT, the minimum service age machine rule, the maximum service age machine rule and random selection, which are marked as S0, S1, S2, S3, S4, S5 and S6 respectively. The 3 maintenance actions are performing a minor repair, performing a medium repair and performing a major repair, which are marked as S7, S8 and S9 respectively:

[0066] 1) Shortest Processing Time (SPPT): Put the processing time corresponding to the set of machines that can be processed in the next process of each workpiece into a list, sort these lists from small to large according to the minimum processing time in each list, and select the process with the shortest processing time for priority processing;

[0067] 2) Longest Processing Time (LPPT): Same as SPPT, the processing time corresponding to the set of machines that can be processed in the next process of each workpiece is put into a list, but these lists are sorted from large to small according to the maximum processing time in each list, and the process with the longest processing time is selected for priority processing;

[0068] 3) SPT (Shortest Processing Time): The shortest processing time rule puts the processing time corresponding to the set of machines that can be processed in the next process of each workpiece into a list, calculates the average processing time in the list, sorts them from small to large according to the average processing time, and selects the workpiece with the shortest average processing time and the earliest idle machine for priority processing;

[0069] 4) LPT (Longest Processing Time): The longest processing time rule is the same as SPT, but the average processing time is sorted from large to small, and the workpiece with the longest average processing time and the earliest idle machine are selected for priority processing;

[0070] 5) Minimum age of the machine rule: Same as SPT, but the machine with the youngest age is given priority for processing;

[0071] 6) Maximum age of the machine rule: Same as LPT, but the machine with the largest age is given priority for processing;

[0072] 7) Random selection: Randomly select an unfinished workpiece and the earliest idle machine in the set of available machines for the next process of the workpiece for processing;

[0073] 8) Perform a minor repair: Select the machine with the longest service life among the idle machines for a minor maintenance. The time required is TM1 and the cost is CM1. The machine status is restored and the service life becomes A. k -D1;

[0074] 9) Perform a medium maintenance: Select the machine with the longest service life among the idle machines for a medium maintenance. The time required is TM2 and the cost is CM2. The machine status is restored and the service life becomes A. k -D2;

[0075] 10) Perform a major overhaul: Select the oldest machine among the idle machines for a major maintenance. The required time is TM3 and the cost is CM3. The machine status is restored and the service life becomes 0.

[0076] The reward mechanism of the present invention is a negative reward, with negative total cost as the reward, because in the environment, the goal of the system is to minimize the total cost. Whenever the total cost increases, the reward will decrease, thereby guiding the DQN algorithm to find a series of actions to minimize the total cost and maximize the cumulative reward in the round. The reward is to return the negative total cost after each round. The more the cost increases, the more negative value -C the reward gets, thereby guiding the agent to find a method with lower cost and setting the reward value reward = -C.

[0077] Furthermore, in step 3, the machine is linearly deteriorated in four stages: in the first deterioration stage, the actual processing time of the workpiece is equal to the original processing time of the workpiece; in the second deterioration stage, the actual processing time of the workpiece is the original processing time plus the time of influence of one deterioration factor; in the third deterioration stage, the actual processing time of the workpiece is the original processing time plus the time of influence of two deterioration factors; in the fourth deterioration stage, the actual processing time of the workpiece is the original processing time plus the time of influence of three deterioration factors, as shown in the formula:

[0078]

[0079] According to the formula for calculating the cost of a single round:

[0080]

[0081] The processing cost of the workpiece will change due to different machine deterioration levels and different actual processing times.

[0082] Furthermore, in step 4, when the machine deteriorates to a high degree, the actual processing time of the workpiece increases dramatically, and the processing cost also increases, which is very unfavorable for the goal of minimizing the total cost. Therefore, every time an action is selected, that is, a workpiece is selected to be processed on the corresponding machine or a machine is selected for maintenance, the deterioration degree of the selected machine is judged. If the machine deteriorates to the fourth stage, that is, the machine age A k When D3 is exceeded, the machine is subject to mandatory maintenance, with maintenance time of TM3 and cost of CM3+C T , the machine status is restored and the service life becomes Ak -D3. Performing mandatory maintenance on older machines can effectively avoid unnecessary processing costs caused by the long service life of the machines. If the machine deteriorates to the first to third stages, the DQN algorithm will explore whether the machine should choose processing or maintenance, and explore the balance between processing and maintenance during learning and training to find the minimum cost.

[0083] Furthermore, in step 5, the information of the workpiece and the machine is updated. Whenever an action is selected, the processing start time, actual processing time and processing end time of the selected workpiece and machine must be confirmed (where the processing start time is the maximum value of the time when the workpiece completes the previous process and the time when the machine completes the previous process, that is, max(CT i ,CT k ), the processing end time is max(CT i ,CT k )+AT ijk , then CT i With CT k Both values ​​are updated as the processing end time). Subsequently, the machine processing time T is updated. k 、The service life of the machine A k , Machine utilization U k , Single round processing cost C s , total cost C and system running time T, where the machine processing time T k 、The service life of the machine A k The update methods of the system running time T are to increase the actual processing time AT ijk , the machine utilization is updated as the ratio of the machine processing time to the system running time:

[0084]

[0085] Finally, the process status of the workpiece is updated. Every time a workpiece completes a process, the process number is increased by 1, and the number of unfinished processes for all workpieces is O. r Subtract 1, and at this time determine whether the number of processes of the current workpiece exceeds the number of processes p that the workpiece should have. If it exceeds, the workpiece is removed from the set UN of unfinished workpieces. i Delete. At the same time, we must also determine the number of remaining processes O r Is it less than or equal to 0? If yes, the processing of all workpieces has been completed and the scheduling is ended; otherwise, continue to select workpieces and machines for processing.

[0086] It should be noted that if the selected action is a processing action, all the above information needs to be updated. If the selected action is a maintenance action, only the maintenance time and service life of the machine need to be updated. k , Single round processing cost Cs , total cost C and system running time T.

[0087] Among them, the maintenance time of the machine is updated as the start maintenance time + maintenance time;

[0088] Single round processing cost C s It is updated to the processing cost or maintenance cost of the current round + the cost of mandatory maintenance;

[0089] The total cost C is updated to be the total cost when the current action is not selected + the single-turn cost of selecting the action.

[0090] Further, in step 6, the DQN algorithm used in the present invention uses a standard neural network structure, including an input layer, a hidden layer and an output layer. The dimension of the input layer is equal to the dimension of the state space; the hidden layer uses a multilayer perceptron (MLP) network, which is composed of two fully connected layers, each layer has 64 neurons, and uses a ReLU (Rectified Linear Unit) activation function; the dimension of the output layer is equal to the dimension of the action space, and the output form is a Q value, and each output value represents the expected total reward for selecting a certain action in a given state. In addition, the Adam optimizer and the mean square error (MSE) loss function are used. The DQN algorithm forms experience by observing the initial state, then selecting an action from the action space in each time step and observing the next state and reward returned by the environment to learn, explore and train. The next state is the average service age, single-round cost, remaining process, average machine utilization and machine deterioration coefficient of all machines mentioned above. After the machine is selected, these data are updated with the update of the state of the selected machine, that is, the next state is obtained.

[0091] Furthermore, in step 7, the recorded scheduling result is the total cost after all workpieces have completed one processing, and the number of training rounds is 600 rounds.

[0092] Furthermore, in step 8, all scheduling results of the four methods, namely, the DQN algorithm, the PPO algorithm, the A2C algorithm, and the random selection scheduling rule, after running for 600 rounds are sorted out and plotted as an iteration graph, with the vertical axis being the total cost after each scheduling, the horizontal axis being the total cost number (number of scheduling completions), and the lowest cost among all total costs being marked. By analyzing the iteration graph, it can be seen whether the algorithm is effective and whether the scheduling is optimized.

[0093] The beneficial effects of the present invention are mainly manifested in:

[0094] 1. The deterioration effect of machines is considered in the hybrid flow shop scheduling problem, and a four-stage linear deterioration function is designed.

[0095] 2. In response to the problem of machine deterioration, three maintenance methods and mandatory maintenance strategies are designed.

[0096] 3. The DQN algorithm is used to solve the dynamic hybrid flow shop scheduling problem considering the deterioration effect, and a new scheduling rule is designed to make the scheduling more efficient. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 The present invention discloses a flow chart of a dynamic hybrid flow shop scheduling method taking deterioration effects into consideration according to an example of the present invention.

[0098] Figure 2 The present invention is a schematic diagram of a processing flow of a hybrid flow shop disclosed in an example of the present invention.

[0099] Figure 3 This is a schematic diagram of a four-stage linear degradation curve function disclosed in an example of the present invention.

[0100] Figure 4 This is a schematic diagram of the DQN algorithm structure disclosed in an example of the present invention.

[0101] Figure 5 It is an iterative schematic diagram of using the DQN algorithm to solve the hybrid flow shop scheduling problem disclosed in an example of the present invention.

[0102] Figure 6 It is an iterative schematic diagram of using the PPO algorithm to solve the hybrid flow shop scheduling problem disclosed in an example of the present invention.

[0103] Figure 7 It is an iterative schematic diagram of using the A2C algorithm to solve the hybrid flow shop scheduling problem disclosed in an example of the present invention.

[0104] Figure 8 It is an iterative schematic diagram of solving a hybrid flow shop scheduling problem using a method of randomly selecting actions disclosed in an example of the present invention. DETAILED DESCRIPTION

[0105] The present invention will be further described below in conjunction with the accompanying drawings.

[0106] refer to Figure 1 ,A dynamic hybrid flow shop scheduling method considering the deterioration effect includes the following steps: building the ,environmental framework of the hybrid flow shop and initializing the ,environmental parameters, defining the state space, action space and reward mechanism, designing the ,deterioration function, setting mandatory maintenance rules, updating information, running the DQN algorithm, recording the ,scheduling results and drawing the iteration graph.

[0107] Step 1: Build the environment framework of the hybrid flow workshop and initialize the environment parameters.

[0108] See Figure 2In this embodiment, there are 10 workpieces, each with 3 processes, and each process has 3 machines in the processable machine set. Each workpiece is processed in a fixed process sequence, and each process can only be processed on a machine in the specified machine set; each machine can only process one process at a time; each process is processed only once, and the next process can only be carried out after the previous process is completed; each process cannot be interrupted during the processing; except for processing, maintenance and waiting time, other time consumption is not considered. The environmental parameters and mathematical symbols involved in this embodiment are as follows:

[0109] n: number of workpieces;

[0110] m: number of machines;

[0111] p: number of workpiece processes;

[0112] r: number of remaining processes;

[0113] J i : the i-th workpiece;

[0114] O ij : The jth process of the i-th workpiece;

[0115] M k : the kth machine;

[0116] M ij :Can process process O ij A collection of machines;

[0117] T ijk :Process O ij On machine M k Processing time on

[0118] Process O ij In M ij Average processing time in

[0119] AT ijk :After the machine deteriorates, process O ij On machine M k The actual processing time on

[0120] CT i : The time when the most recent process of workpiece i is completed;

[0121] CT k : The time when the machine last completed processing;

[0122] S i : The process that workpiece i will need to be processed;

[0123] Or : The number of unfinished processing steps for all workpieces;

[0124] UN i : A collection of unfinished workpieces;

[0125] T k : Machine processing time;

[0126] A k : Machine age (after maintenance operations, the machine age will be restored to a certain extent);

[0127] A ave : Average age of all machines;

[0128] U k : Machine utilization;

[0129] U ave : The average utilization of all machines;

[0130] d: deterioration factor;

[0131] D a : The age of the machine at the turning point of machine deterioration, a=1, 2 or 3; where D1, D2 and D3 represent the maximum age of the machine in the first, second and third deterioration stages respectively. When the machine age reaches and exceeds the corresponding deterioration turning point, the machine enters a new deterioration stage;

[0132] D k : Machine deterioration coefficient;

[0133] mt a : The number of maintenance times for each type of maintenance, a=1,2 or 3; where mt1, mt2 and mt3 represent the number of maintenance times for minor repairs, medium repairs and major repairs respectively;

[0134] CM a : The cost of each type of maintenance, a = 1, 2 or 3; CM1, CM2 and CM3 represent the cost of a minor repair, a medium repair and a major repair respectively;

[0135] TM a : The time required for each type of maintenance, a = 1, 2 or 3; TM1, TM2 and TM3 represent the time required for a minor repair, a medium repair and a major repair respectively;

[0136] mq: the number of times the mandatory maintenance is performed;

[0137] C t : Machine operation cost per unit time;

[0138] C s: The unit cost after each processing step or each maintenance completed;

[0139] C: total cost;

[0140] T: system running time;

[0141]

[0142] The goal of this embodiment is to minimize the total cost. The machine operation cost per unit time is 30 yuan; the minor repair time is 5 hours, the cost is 100 yuan, and the machine service life is reduced by 10 hours; the medium repair time is 9 hours, the cost is 180 yuan, and the machine service life is reduced by 30 hours; the major repair time is 15 hours, the cost is 300 yuan, and the machine service life is restored to 0. The additional maintenance cost of mandatory maintenance of the machine is 300 yuan.

[0143] The goal of workshop scheduling is to minimize the total cost, which can be expressed in the following formula:

[0144]

[0145] Step 2: Define the state space, action space and reward mechanism. Based on the observation and analysis of the actual production environment, this embodiment extracts 5 state features from the workpiece and machine state transitions, and designs and selects 10 actions.

[0146] The state characteristics are the average age of all machines, the cost per round, the remaining processes, the average utilization rate of machines and the machine deterioration coefficient. The action space includes ten actions, including 7 processing actions and 3 maintenance actions. The 7 processing actions are the shortest processing time rule, the longest processing time rule, SPT, LPT, the minimum age machine rule, the maximum age machine rule and random selection, marked as S0, S1, S2, S3, S4, S5 and S6 respectively. The 3 maintenance actions are to perform a minor repair, a medium repair and a major repair, marked as S7, S8 and S9 respectively.

[0147] The reward is a negative reward, with a negative total cost as the reward, because in the environment, the goal of the system is to minimize the total cost. Whenever the total cost increases, the reward will decrease, thus guiding the DQN algorithm to find a series of actions that minimize the total cost and maximize the cumulative reward in the round. The formula for the state is as follows:

[0148] (1) Average age of all machines:

[0149]

[0150] A k (t) represents the machine age at the current time t;

[0151] (2) Single round cost:

[0152]

[0153] (3) Remaining processes: After each process is completed, the number of remaining processes r decreases by 1.

[0154] (4) Average machine utilization rate:

[0155]

[0156] U k (t) represents the machine utilization at the current time t;

[0157] (5) Machine deterioration coefficient:

[0158]

[0159] Step 3: Design a deterioration function to calculate the processing time and cost of the machine under different deterioration levels. As the machine processing time increases, the state of the machine will also change, such as tool wear, part failure, etc., which can be called machine deterioration. When the machine deteriorates, the state of the machine when processing the workpiece will affect the processing time, so it is necessary to consider the impact of machine deterioration on processing costs.

[0160] See Figure 3 This embodiment observes and analyzes the actual production environment, assumes the deterioration factor of the machine and the four-stage linear deterioration function, sets the deterioration factor to 0.3, and has three deterioration turning points, which are 10 hours, 30 hours, and 60 hours of machine service life. As the machine deteriorates, the actual processing time changes as shown in the formula:

[0161]

[0162] Calculate the cost per round:

[0163]

[0164] Step 4: Set mandatory maintenance rules. When the machine deteriorates to a certain extent, mandatory maintenance is performed. When the machine deteriorates to a high degree, the actual processing time of the workpiece increases dramatically, and the processing cost also increases dramatically, which is very unfavorable for the goal of minimizing the total cost. Therefore, every time a workpiece is selected to be processed on the corresponding machine or a machine is selected for maintenance, the selected machine will be judged for its deterioration degree. If the machine deteriorates to the fourth stage, that is, the machine age A k When the service life exceeds 60 hours, the machine will be subject to mandatory maintenance, which takes 15 hours and costs 600 yuan. The machine status will be restored and the service life will be changed to A. k-60. Mandatory maintenance of older machines can effectively avoid unnecessary processing costs caused by the long service life of the machines.

[0165] Step 5: Update the relevant information of the workpiece and the machine. Since scheduling needs to be based on the processing time of the workpiece and the machine, the status of each workpiece and each machine needs to be updated after the processing is completed to ensure the correctness of the scheduling information.

[0166] First, update the processing start time, actual processing time and processing end time of the selected workpiece and machine. The processing start time is the maximum value of the time when the workpiece completes the previous process and the time when the machine completes the previous process, and the processing end time is the processing start time plus the actual processing time; then, update the machine processing time T k 、The service life of the machine A k , Machine utilization U k , Single round processing cost C s , total cost C and system running time T, where the machine processing time T k 、The service life of the machine A k The update methods of the system running time T are to increase the actual processing time AT ijk Finally, update the process status of the workpiece. Every time a workpiece completes a process, the process number is increased by 1, and the number of unfinished processes for all workpieces is O. r Subtract 1, and at this time determine whether the number of processes of the current workpiece exceeds the number of processes that the workpiece should have. That is, in this embodiment, if the number of processes of the current workpiece exceeds 3, the workpiece is removed from the set UN of unfinished workpieces. i Delete. At the same time, we must also determine the number of remaining processes O r Is it less than or equal to 0? If yes, the processing of all workpieces has been completed and the scheduling is ended; otherwise, continue to select workpieces and machines for processing.

[0167] It should be noted that if the selected action is a processing action, all the above information needs to be updated. If the selected action is a maintenance action, only the maintenance time and service life of the machine need to be updated. k , Single round processing cost C s , total cost C and system running time T.

[0168] The maintenance time of the machine is updated to the start maintenance time + maintenance time;

[0169] Single round processing cost C s Updated to the processing cost or maintenance cost of the current round + the cost of mandatory maintenance;

[0170] The total cost C is updated to the total cost when the current action is not selected + the single-round cost of selecting this action;

[0171] Machine utilization U k Updated to the ratio of machine processing time to system running time:

[0172]

[0173] Step 6: Run the DQN algorithm for learning and training.

[0174] See Figure 4 , the DQN algorithm used in this embodiment uses a standard neural network structure, including an input layer, a hidden layer, and an output layer. The dimension of the input layer is equal to the dimension of the state space, and the input value is 5 state values; the hidden layer uses a multilayer perceptron (MLP) network, which consists of two fully connected layers, each with 64 neurons, and uses the ReLU (Rectified Linear Unit) activation function; the dimension of the output layer is equal to the dimension of the action space, and the output form is the Q value of each of the 10 actions. The action is selected according to the Q value and executed in the environment to obtain the current reward and the next state. Then, after a simulation experiment on a hybrid flow workshop, the learning rate is selected as 0.01 and the discount rate is 0.99. In addition, the Adam optimizer and the mean square error (MSE) loss function are also used. The Adam optimizer automatically adjusts the learning rate according to the gradient information to accelerate convergence. It maintains the average gradient and square gradient of each parameter for calculating the adaptive learning rate. The mean square error (MSE) loss function is used to calculate the gap between the current Q value (predicted Q value) and the target Q value.

[0175] Step 7: Record the scheduling results and determine whether the training time has reached the termination condition. After a certain amount of experiments, choose to learn and train the algorithm for 600 rounds.

[0176] Step 8: Draw the total cost iteration graph, compare the DQN algorithm with other algorithms, and analyze the scheduling results.

[0177] See Figures 5 to 8 After running each method for about 600 rounds, draw a total cost iteration graph, with the vertical axis being the total cost after each scheduling completion, and the horizontal axis being the total cost number (number of scheduling completions), and mark the lowest cost among all total costs. Figure 5 It is the DQN algorithm. Figure 6 is the PPO algorithm, Figure 7 It is the A2C algorithm. Figure 8 is a random selection of actions, through Figures 5 to 8It can be seen that after running for about 600 rounds, the DQN algorithm found the minimum total cost of 9168.6 yuan, the PPO algorithm found it to be 10327.9 yuan, the A2C algorithm found it to be 9360.84 yuan, and the random selection scheduling rule method found it to be 12995.97 yuan. It is obvious that the DQN algorithm is more effective in solving the problem in this embodiment, and can significantly reduce the total cost while using less algorithm running time.

[0178] This embodiment discloses a dynamic hybrid assembly line scheduling method that takes deterioration effects into consideration. The method uses pycharm software, and uses the gym and stable-baselines3 packages to implement a deep learning model. The deep reinforcement learning model can be run in pycharm to implement a dynamic hybrid assembly line scheduling system that takes deterioration effects into consideration. The main steps in the process are to build a hybrid assembly line scheduling environment, design deterioration functions, design scheduling rules, and run the DQN algorithm to solve the problem. All of these are implemented on the pychram platform, and the present invention will not go into detail. The main function of the present invention is to propose a dynamic hybrid assembly line scheduling method that takes deterioration effects into consideration, and uses a parameter simulation method to provide the degree of fit between the hybrid assembly line model and the real processing space in terms of machine deterioration and maintenance. In addition, Figure 5-Figure 8 The comparison of iteration graphs proves that a dynamic hybrid flow shop scheduling method considering the deterioration effect can effectively reduce the total cost of workshop scheduling, effectively solve the scheduling problem of hybrid flow shops, and be applied in practical fields.

[0179] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

[0180] In conclusion, the above is only a preferred embodiment of the present invention, and all equivalent changes and modifications made according to the scope of the patent application of the present invention should fall within the scope of the patent of the present invention.

Claims

1. A dynamic hybrid flow shop scheduling method considering deterioration effect, characterized in that The following steps are involved: Step 1: Build the environment framework of the hybrid flow shop and initialize the environment parameters, set the constraints of the shop scheduling, and set the scheduling goal to minimize the total cost; Step 2: Define the state space, action space and reward mechanism; Step 3: Design a deterioration function to calculate the processing time and cost of the machine under different deterioration levels; Step 4: Set mandatory maintenance rules. When the machine deteriorates to a certain extent, mandatory maintenance will be performed. Step 5: Update the relevant information of the workpiece and machine; Step 6: Run the DQN algorithm for learning and training; Step 7: Record the scheduling results and determine whether the training round has reached the termination condition. If so, execute step 8, otherwise return to step 6; Step 8: Draw a total cost iteration graph, with the vertical axis representing the total cost after each scheduling completion and the horizontal axis representing the number of scheduling completions. Analyze the scheduling results based on the iteration graph to obtain the workshop scheduling result with the lowest cost.

2. A dynamic hybrid flow shop scheduling method considering deterioration effect as described in claim 1, characterized in that In step 1, there are n workpieces and m machines in the design workshop. Each workpiece has the same p processes. Each process has a machine set, which consists of machines that can process the process. In the same machine set, the processing time of each machine for the same process of the same workpiece is not exactly the same. At the same time, the workshop also has some constraints: each workpiece is processed in a fixed process sequence, and each process can only be processed on machines in the specified machine set; each machine can only handle one process at a time; each process is processed only once, and the next process can only be carried out after the previous process is completed; each process cannot be interrupted during the processing; except for processing, maintenance and waiting time, other time consumption is not considered; The environmental parameters include the following: n: number of workpieces; m: number of machines; p: number of workpiece processes; r: the number of remaining processes of the workpiece; J i : the i-th workpiece; O ij : The jth process of the i-th workpiece; M k : the kth machine; M ij :Can process process O ij A collection of machines; T ijk :Process O ij On machine M k Processing time on Process O ij In M ij Average processing time in AT ijk :After the machine deteriorates, process O ij On machine M k The actual processing time on CT i : The time when the most recent process of workpiece i is completed; CT k : The time when the machine last completed processing; S i : The process that workpiece i will need to be processed; O r : The number of unfinished processing steps for all workpieces; UN i : A collection of unfinished workpieces; T k : Machine processing time; A k : The service life of the machine will be restored to a certain extent after maintenance operations; A ave : Average age of all machines; U k : Machine utilization; U ave : The average utilization of all machines; d: deterioration factor; D a : The age of the machine at the turning point of machine deterioration, a=1, 2 or 3; where D1, D2 and D3 represent the maximum age of the machine in the first, second and third deterioration stages respectively. When the machine age reaches and exceeds the corresponding deterioration turning point, the machine enters a new deterioration stage; D k : Machine deterioration coefficient; m t : The number of maintenance times for each type of maintenance, a=1,2 or 3; where mt1, mt2 and mt3 represent the number of maintenance times for minor repairs, medium repairs and major repairs respectively; CM a : The cost of each type of maintenance, a = 1, 2 or 3; CM1, CM2 and CM3 represent the cost of a minor repair, a medium repair and a major repair respectively; TM a : The time required for each type of maintenance, a = 1, 2 or 3; TM1, TM2 and TM3 represent the time required for a minor repair, a medium repair and a major repair respectively; mq: the number of times the mandatory maintenance is performed; C t : Machine operation cost per unit time; C s : The unit cost after each processing step or each maintenance completed; C: total cost; T: system running time; The goal of workshop scheduling is to minimize the total cost, which can be expressed in the following formula: The C T It represents the additional repair cost of mandatory maintenance on the machine. The mandatory maintenance cost of the machine = the cost required for an overhaul + the additional repair cost.

3. A dynamic hybrid flow shop scheduling method considering deterioration effect as claimed in claim 1, characterized in that In step 2, the state space includes five state values, namely, the average service life of all machines, the cost per round, the remaining processes, the average utilization rate of machines, and the machine deterioration coefficient: (1) Average age of all machines: A k (t) represents the machine age at the current time t; (2) Single round cost: (3) Remaining processes: After each process is completed, the number of remaining processes r decreases by 1; (4) Average machine utilization rate: U k (t) represents the machine utilization at the current time t; (5) Machine deterioration coefficient: The action space includes ten actions, including 7 processing actions and 3 maintenance actions. The 7 processing actions are the shortest processing time rule, the longest processing time rule, SPT, LPT, the minimum service age machine rule, the maximum service age machine rule and random selection, which are marked as S0, S1, S2, S3, S4, S5 and S6 respectively. The 3 maintenance actions are performing a minor repair, performing a medium repair and performing a major repair, which are marked as S7, S8 and S9 respectively: 1) The shortest processing time rule for a process: put the processing time corresponding to the set of machines that can be processed in the next process of each workpiece into a list, sort these lists from small to large according to the minimum processing time in each list, and select the process with the shortest processing time for priority processing; 2) The longest processing time rule of a process: put the processing time corresponding to the set of machines that can be processed in the next process of each workpiece into a list, sort these lists from large to small according to the maximum processing time in each list, and select the process with the longest processing time for priority processing; 3) SPT: Shortest processing time rule: put the processing time corresponding to the set of machines that can be processed in the next process of each workpiece into a list, calculate the average processing time in the list, sort them from small to large according to the average processing time, and select the workpiece with the shortest average processing time and the earliest idle machine for priority processing; 4) LPT: The longest processing time rule, which puts the processing time corresponding to the set of machines that can be processed in the next process of each workpiece into a list, calculates the average processing time in the list, sorts them from large to small according to the average processing time, and selects the workpiece with the longest average processing time and the earliest idle machine for priority processing; 5) Minimum age machine rule: put the processing time corresponding to the set of machines that can be processed in the next process of each workpiece into a list, and select the workpiece with the shortest average processing time and the machine with the shortest age to process it first; 6) Rule for the oldest machine: put the processing time corresponding to the set of machines that can be processed in the next process of each workpiece into a list, and select the workpiece with the longest average processing time and the machine with the longest service life to process it first; 7) Random selection: Randomly select an unfinished workpiece and the earliest idle machine in the set of available machines for the next process of the workpiece for processing; 8) Perform a minor repair: Select the machine with the longest service life among the idle machines for a minor maintenance. The time required is TM1 and the cost is CM1. The machine status is restored and the service life becomes A. k -D1; 9) Perform a medium maintenance: Select the machine with the longest service life among the idle machines for a medium maintenance. The time required is TM2 and the cost is CM2. The machine status is restored and the service life becomes A. k -D2; 10) Perform a major overhaul: Select the machine with the longest service life among the idle machines for a major maintenance. The required time is TM3 and the cost is CM3. The machine status is restored and the service life becomes 0. The reward mechanism is a negative reward, with a negative total cost as the reward, because in the environment, the goal of the system is to minimize the total cost. Whenever the total cost increases, the reward will decrease, thereby guiding the DQN algorithm to find a series of actions that minimize the total cost and maximize the cumulative reward in the round.

4. A method for scheduling a dynamic hybrid flow shop considering deterioration effects as claimed in claim 1, characterized in that In step 3, the machine is linearly deteriorated in four stages: in the first deterioration stage, the actual processing time of the workpiece is equal to the original processing time of the workpiece; In the second deterioration stage, the actual processing time of the workpiece is the original processing time plus the time affected by the deterioration factor; In the third deterioration stage, the actual processing time of the workpiece is the original processing time plus the time of twice the deterioration factor; in the fourth deterioration stage, the actual processing time of the workpiece is the original processing time plus the time of three times the deterioration factor, as shown in the formula: According to the formula for calculating the cost of a single round: The processing cost of the workpiece will change due to different machine deterioration levels and different actual processing times.

5. A dynamic hybrid flow shop scheduling method considering deterioration effect as claimed in claim 4, characterized in that The process of the mandatory maintenance rule in step S4 is: every time an action is selected, that is, a workpiece is selected to be processed on the corresponding machine or a machine is selected for maintenance, the deterioration degree of the selected machine will be judged. If the machine deteriorates to the fourth stage, that is, the machine age A k When D3 is exceeded, the machine is subject to mandatory maintenance, with maintenance time of TM3 and cost of CM3+C T , the machine status is restored and the service life becomes A k -D3.

6. A method for scheduling a dynamic hybrid flow shop considering deterioration effects as claimed in claim 1, characterized in that In step 5, the process of updating the information of the workpiece and the machine includes: Whenever an action is selected, the processing start time, actual processing time and processing end time of the selected workpiece and machine must be confirmed. The processing start time is the maximum value of the time when the workpiece completes the previous process and the time when the machine completes the previous process, that is, max(CT i ,CT k ), the processing end time is max(CT i ,CT k )+AT ijk , then CT i With CT k Both values ​​are updated to the processing end time; If the selected action is a maintenance action, you only need to update the machine's maintenance time and the machine's service life. k , Single round processing cost C s , total cost C and system running time T; if the selected action is a processing action, then update the machine processing time T k 、The service life of the machine A k , Machine utilization U k , Single round processing cost C s , total cost C and system running time T; The machine has been processing for T k 、The service life of the machine A k The update methods of the system running time T are to increase the actual processing time AT ijk ; The maintenance time of the machine is updated as the start maintenance time + maintenance time; Single round processing cost C s It is updated to the processing cost or maintenance cost of the current round + the cost of mandatory maintenance; The total cost C is updated to the total cost when the current action is not selected + the single-round cost of selecting this action; Machine utilization U k Updated to the ratio of machine processing time to system running time: Finally, the process status of the workpiece is updated. Every time a workpiece completes a process, the process number is increased by 1, and the number of unfinished processes for all workpieces is O. r Subtract 1, and at this time determine whether the number of processes of the current workpiece exceeds the number of processes p that the workpiece should have. If it exceeds, the workpiece is removed from the set UN of unfinished workpieces. i Delete; at the same time, we must also determine the number of remaining processes O r Is it less than or equal to 0? If so, the processing of all workpieces has been completed and the scheduling is ended. Otherwise, continue to select workpieces and machines for processing.

7. A dynamic hybrid flow shop scheduling method considering deterioration effect as claimed in claim 3, characterized in that In step 6, the DQN algorithm forms experience by observing the initial state, then selecting an action from the action space in each time step and observing the next state and reward returned by the environment, thereby learning, exploring and training; the DQN algorithm is a neural network structure, including an input layer, a hidden layer and an output layer, the dimension of the input layer is equal to the dimension of the state space, the hidden layer uses a multi-layer perceptron MLP network, which is composed of two fully connected layers, each layer has 64 neurons, and uses a ReLU activation function; the dimension of the output layer is equal to the dimension of the action space, and the output is in the form of a Q value, and each output Q value represents the expected total reward for selecting a certain action in a given state.

Images