Flexible job-shop scheduling method based on deep reinforcement learning and fused with multi-head attention mechanism

Abstract

The invention relates to the technical field of flexible job shop scheduling, and discloses a deep reinforcement learning-based flexible job shop scheduling method fused with a multi-head attention mechanism. The method comprises the following steps of: obtaining initial feature information of all workpieces in a current environment according to a flexible job shop environment at the current moment; the scheduling action is obtained according to the initial feature information of all machines in the current environment, the initial feature information of the process-machine pair in the current environment, the feature information of the action space in the current environment and the feature information of the next scheduling moment in the current environment, and whether the scheduling is completed or not is judged. According to the method, the FJSSP is modeled based on deep reinforcement learning and through the Markov continuous decision process concept, so that the quality of solving the FJSP solution by using the DRL is obviously improved, and meanwhile, the action to be executed can be more accurately and remotely judged more comprehensively through the proposed new state extraction network and the DRL model, so that the training is more efficient.

Description

Technical Field

[0001] The present invention relates to the technical field of flexible job shop scheduling, and specifically to a flexible job shop scheduling method based on deep reinforcement learning and integrating a multi-head attention mechanism. Background Art

[0002] With the globalization of the economy and the continuous development of the intelligent manufacturing industry, the production process of products has become more refined and the production of products has become more flexible. This means that there are multiple processing schemes for the same product. And for each batch of products, since their processing goals are not necessarily the same, the selected processing schemes are not necessarily the same. For each processing scheme, when it is refined to the factory, it is a workshop scheduling problem. The traditional workshop scheduling problem is mainly concentrated in one factory. At the same time, it is divided into flow workshop scheduling problems, flexible workshop scheduling problems, etc. In most cases, flexible workshop scheduling problems are the most common and efficient;

[0003] Solving FJSP usually includes two parts: 1. Machine selection for product processing procedures (machine selection problem), 2. Processing order of product procedures on each machine (sequencing problem). There are many ways to solve FJSP. One is to obtain an exact solution. Common ways to obtain an exact solution include linear programming, dynamic programming, and branch-and-bound algorithms. However, since FJSP is NP-hard, it takes a long time to use these methods and it is not easy to obtain a solution. The other is to obtain an approximate solution. Currently, approximate solutions can be roughly divided into three categories: (1) Scheduling rules. The simplest method is to apply scheduling rules to assign different priorities to all jobs waiting to be processed on the machine. When the machine is available, the job with the highest priority is selected for processing. Some commonly used dispatching rules include "first in, first out", "shortest processing time", "longest processing time", "earliest due date" and the composition of these rules. These scheduling rules are often difficult to obtain satisfactory scheduling in practical applications. (2) Heuristic search methods. These methods reduce the search space of scheduling solutions based on heuristic algorithms, which rank alternatives at each branch step, decide which branch to follow, and stop branch search at any time if the current possibility is worse than the current solution. Popular heuristic algorithms include taboo search, simulated annealing, and evolutionary algorithms. These heuristic search methods can effectively reach possible solutions, but they are usually trapped in local optima. (3) Reinforcement learning (RL) methods. The algorithm framework based on RL is to model the factory as an environment according to the RL idea, where the state consists of various characteristics of machines and jobs, and the action refers to the decision of job sequencing and routing. The agent learning embedded in the RL method maps each state to the best action in order to maximize the return of the sum of discounted rewards obtained from the environment. However, except for using reinforcement learning to solve the FJSP problem, the other solutions mentioned above have certain limitations. The time cost of the exact solution type algorithm is too large; the traditional scheduling rule algorithm is not easy to find the optimal solution; the stability of the quality of the heuristic search solution cannot be guaranteed because the method often uses intelligent optimization algorithms, which have random factors in such algorithms, and the design of these methods is also relatively complex. The time cost of using RL to solve FJSP is much less than that of other methods, and it can effectively avoid falling into the dilemma of local optimal solution. Therefore, in recent years, most scholars have begun to combine RL to solve various JSP problems. Undoubtedly, using RL to solve FJSP is efficient and reasonable.

[0004] To solve FJSP using DRL, we first need to transform a scheduling problem into a process that requires continuous decision-making. In most methods that use DRL to solve FJSP, the scheduling process is usually modeled as an MDP. In these methods, a parametric neural network model is designed to receive information about the production environment as a state and output the priority of each feasible scheduling action, such as assigning operations to machines, forming an end-to-end learning method. By training on a set of production process data, the DRL model learns to adaptively select the best action in a state to maximize the total reward associated with the production goal. However, the effectiveness and efficiency of these methods rely heavily on the design of state representation, which is a challenging task. As the size of the problem increases, the amount of production information increases significantly. Therefore, in addition to the problem of how to cleverly transform the scheduling problem into an MDP, another key challenge is how to design a sufficient and minimal representation to represent the environmental information in the MDP process and the efficient processing of this information.

[0005] At present, the research on solving FJSSP using DRL method is still in its infancy. For the extraction of environmental information in the MDP process, the main methods currently used are multi-layer perceptron (MLP), neural networks under isomorphic or heterogeneous graphs, and more detailed feature definitions to extract information about the system environment, but these methods cannot well associate local features with overall features; at the same time, most of the modeling of FJSP is not detailed enough; this makes the DRL model not comprehensive enough for the overall input information of the problem, and some defects in the modeling method make the quality of the solution generally insufficient. Summary of the invention

[0006] 1. Technical issues to be resolved

[0007] In view of the shortcomings of the prior art, the present invention provides a flexible job shop scheduling method based on deep reinforcement learning and integrated with a multi-head attention mechanism, which has the advantages of efficient scheduling and solves the above-mentioned technical problems.

[0008] (II) Technical solution

[0009] To achieve the above object, the present invention provides the following technical solution: a flexible job shop scheduling method based on deep reinforcement learning and fusion of multi-head attention mechanism, comprising the following steps:

[0010] S1. According to the current flexible workshop environment, obtain the initial feature information of all workpieces in the current environment Initial feature information of all machines in the current environment Initial feature information of process-machine pairs in the current environment Feature information of the action space in the current environment And the characteristic information μ of the next scheduling time of the current environment t ;

[0011] Among them, for the initial feature information of the workpiece: They represent the release time of workpiece J1 at the current scheduling time t, the release time of workpiece J2 at the current scheduling time t, ..., the release time of workpiece J at the current scheduling time t n The release time of J1, J2, ..., J n represents the operation of workpiece i, They represent the time difference of the release time of the same workpiece J1 at the current scheduling time t, the time difference of the release time of the same workpiece J2 at the current scheduling time t, ..., the time difference of the release time of the same workpiece J at the current scheduling time t n If the release time of the workpiece is less than or equal to the current scheduling time, the difference is 0. They represent the average processing time of the pending processing steps of workpiece J1 at the current scheduling time t, the average processing time of the pending processing steps of workpiece J2 at the current scheduling time t, ..., the average processing time of the pending processing steps of workpiece J at the current scheduling time t n The average processing time of the processing steps to be processed, They represent the minimum processing time of the pending processing procedure of workpiece J1 at the current scheduling time t, the minimum processing time of the pending processing procedure of workpiece J2 at the current scheduling time t, ..., the minimum processing time of the pending processing procedure of workpiece J at the current scheduling time t n The minimum processing time of the processing steps to be processed, They represent the remaining scheduled processes of workpiece J1 at the current scheduling time t, the remaining scheduled processes of workpiece J2 at the current scheduling time t, ..., the remaining scheduled processes of workpiece J at the current scheduling time t n The remaining processes to be scheduled;

[0012] For the initial feature information of the machine: They represent the number of processes for processing and machine M1 at the current scheduling time t, the number of processes for processing and machine M2 at the current scheduling time t, and the number of processes for processing and machine M at the current scheduling time t. k The number of processes, They represent the release time of processing and machine M1 at the current scheduling time t, the release time of processing and machine M2 at the current scheduling time t, ..., the release time of processing and machine M at the current scheduling time t k The release time, They represent the difference between the release time of processing and machine M1 at the current scheduling time t, the difference between the release time of processing and machine M2 at the current scheduling time t, and the difference between the release time of processing and machine M at the current scheduling time t. k The difference in release time, if the mth machine M mIf the release time of is less than or equal to the current scheduling time t, the difference is 0;

[0013] For the initial feature information of the process-machine pair: They represent the process O at the current scheduling time t. 1,j , O 2, j, ..., O n,j Machining time on machine M1, They represent the process O at the current scheduling time t. 1,j , O 2,j ,…,O n,j On machine M k The processing time on They represent the process O at the current scheduling time t. 1,j , O 2,j ,…,O n,j The earliest start time of processing on machine M1, They represent the process O at the current scheduling time t. 1,j , O 2,j ,…,O n,j On machine M k The earliest processing start time, Respectively represent the current combination process O 1,j , O 2,j ,…,O n,j The minimum processing time is the same as the current combined process O 1,j -M1, O 2,j -M1, ..., O n,j -Difference in processing time of M1, Respectively represent the current combination process O 1,j , O 2,j ,…,O n,j The minimum processing time is the same as the current combined process O 1,j -M k , O 2,j -M k ,…,O n,j -M k The difference in processing time;

[0014] For action space features: It represents the availability of the entire workpiece at the current scheduling time t, which is calculated by dividing the current available number of workpieces by the total number of workpieces. Indicates the availability of all machines at the current scheduling time, which is calculated by dividing the current number of available machines by the total number of machines. Indicates the average release time of the workpiece in the current state. The calculation method is the sum of the release time of all current workpieces divided by the total number of workpieces. Indicates the average release time of the machine in the current state. It is calculated by dividing the sum of the release times of all current machines by the total number of machines. It indicates the ratio of the optimal solution process that can be obtained in the current state, which is calculated by dividing the number of all the processes to be processed that can be processed on the machine with the minimum processing time by the total number of workpieces;

[0015] For the next scheduling time feature: μ t Indicates the time difference between the current scheduling time and the next scheduling time;

[0016] S2. Initial feature information of all workpieces in the current environment And the initial feature information of all machines in the current environment The input image is fed into a multi-head self-attention network to obtain new workpiece features and new machine features, which are then pooled separately to obtain pool feats.

[0017] S3: Pool feats obtained by pooling and feature information of action space in the current environment And the next scheduling time characteristic μ t Splice and get the final information of the machine process pair, that is, the status Combine new workpiece features and new machine features with the initial feature information of the process-machine pair in the current environment Splice to get the final information of the action space, that is, the state

[0018] S4, the state Input action space agent, through the probability distribution of whether to transfer action space generated by action space agent, get action according to sampling or greedy strategy The status Enter the action agent, through the action probability distribution generated by the action agent, and obtain the action according to the sampling or greedy strategy

[0019] S5. According to the action Determine whether to perform action space transfer. If so, perform action space transfer and get reward If no, then perform the action Get rewarded

[0020] S6. Determine whether all scheduling has been completed. If yes, end the process; if not, return to step S1.

[0021] As a preferred technical solution of the present invention, the reward is the difference between the next scheduling time and the current time, and its specific expression is as follows:

[0022]

[0023] Among them, Time t Indicates the current time, Time t+1 Indicates the next scheduling time;

[0024] The Reward use Estimate a completion time Reuse Estimate the next completion time The expression for the final reward is as follows:

[0025]

[0026] Among them, C msx (S t ) represents the completion time, C max (S t+1 ) indicates the next completion time.

[0027] Compared with the prior art, the present invention provides a flexible job shop scheduling method based on deep reinforcement learning and fusion of multi-head attention mechanism, which has the following beneficial effects:

[0028] The present invention significantly improves the quality of FJSP solutions using DRL by modeling FJSP based on deep reinforcement learning and the concept of Markov continuous decision process. At the same time, through the proposed new state extraction network, the DRL model can make more comprehensive, accurate and long-term judgments on the actions to be performed. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] Figure 1 It is a schematic diagram of the overall process of the present invention;

[0030] Figure 2 A schematic diagram of the network input structure of the present invention;

[0031] Figure 3 It is a schematic diagram of the experimental effect of the present invention;

[0032] Figure 4 Schematic diagram of the solution of Example 18a in the Dauzere test set of the present invention. Figure 5 It is a schematic diagram of obtaining new features in step S2 of the present invention. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0034] FJSSP definition: There are n workpieces J = {J1, J2, ..., J n} requires k available machines M = {M1,M2,…,M k Each workpiece contains at least one process, and the adjacent processes of the same workpiece must satisfy the priority constraint. i,j , only one machine can be selected from the optional machine set to process the process, and the processing time of the machines in the optional machine set is different. Under the condition of satisfying various constraints, namely priority constraints and capacity constraints, the optimization goal of the present invention is to maximize the completion time C max minimize;

[0035] The expression of the FJSSP model of the present invention is as follows:

[0036]

[0037] Where i and i′ represent different workpiece indices, j and j′ represent different operation indices, k represents the number of machines, J = {1, 2, …, n} represents the workpiece set, and J i represents the set of operations for workpiece i, M = {1, 2, ..., k} represents the set of machines, M i,j Indicates that process O can be i,j The collection of machines that perform processing, M i,ni represents the last process n of workpiece i i The number of machines that can process, n represents the number of workpieces, n i represents the number of processes of workpiece i, m represents the machine index, L represents a very large integer, and O i,j represents the jth process of workpiece i, B i,j Indicates process O i,j Start time, B i′,j′ Process O i′,j′ Start time, pt i,j,m Indicates process O i,j Processing time on machine m, pt i′,j′,m Indicates process O i′,j′ Processing time on machine m, Indicates process n of process i i Processing time on machine m, represents the process n of workpiece ii Can it be processed on machine m? When process O i,j If it can be processed on machine m, then X i,j,m =1, otherwise X i,j,m =0, When process O i,j In process O i′,j′ Before processing, Y i,j,i′,j′ =1, otherwise Y i,j,i′,j′ =0.

[0038] See also Figure 1-Figure 4 , a flexible job shop scheduling method based on deep reinforcement learning and fusion of multi-head attention mechanism, including the following steps:

[0039] S1. According to the current flexible workshop environment, obtain the initial feature information of all workpieces in the current environment Initial feature information of all machines in the current environment Initial feature information of process-machine pairs in the current environment Feature information of the action space in the current environment And the characteristic information μ of the next scheduling time of the current environment t ;

[0040] Among them, for the initial feature information of the workpiece: They represent the release time of workpiece J1 at the current scheduling time t, the release time of workpiece J2 at the current scheduling time t, ..., the release time of workpiece J at the current scheduling time t n The release time of J1, J2, ..., J n represents the operation of workpiece i, They represent the time difference of the release time of the same workpiece J1 at the current scheduling time t, the time difference of the release time of the same workpiece J2 at the current scheduling time t, ..., the time difference of the release time of the same workpiece J at the current scheduling time t n If the release time of the workpiece is less than or equal to the current scheduling time, the difference is 0. They represent the average processing time of the pending processing steps of workpiece J1 at the current scheduling time t, the average processing time of the pending processing steps of workpiece J2 at the current scheduling time t, ..., the average processing time of the pending processing steps of workpiece J at the current scheduling time t n The average processing time of the processing steps to be processed, They represent the minimum processing time of the pending processing procedure of workpiece J1 at the current scheduling time t, the minimum processing time of the pending processing procedure of workpiece J2 at the current scheduling time t, ..., the minimum processing time of the pending processing procedure of workpiece J at the current scheduling time t n The minimum processing time of the processing steps to be processed, They represent the remaining scheduled processes of workpiece J1 at the current scheduling time t, the remaining scheduled processes of workpiece J2 at the current scheduling time t, ..., the remaining scheduled processes of workpiece J at the current scheduling time t n The remaining processes to be scheduled;

[0041] For the initial feature information of the machine: They represent the number of processes for processing and machine M1 at the current scheduling time t, the number of processes for processing and machine M2 at the current scheduling time t, and the number of processes for processing and machine M at the current scheduling time t. k The number of processes, They represent the release time of processing and machine M1 at the current scheduling time t, the release time of processing and machine M2 at the current scheduling time t, ..., the release time of processing and machine M at the current scheduling time t k The release time, They represent the difference between the release time of processing and machine M1 at the current scheduling time t, the difference between the release time of processing and machine M2 at the current scheduling time t, and the difference between the release time of processing and machine M at the current scheduling time t. k The difference in release time, if the mth machine M m If the release time of is less than or equal to the current scheduling time t, the difference is 0;

[0042] For the initial feature information of the process-machine pair: They represent the process O at the current scheduling time t. 1,j , O 2,j ,…,O n,j Machining time on machine M1, They represent the process O at the current scheduling time t. 1,j , O 2,j ,…,O n,j On machine M k The processing time on They represent the process O at the current scheduling time t. 1,j , O 2,j ,…,O n,j The earliest start time of processing on machine M1, They represent the process O at the current scheduling time t. 1,j , O 2,j ,…,O n,j On machine M k The earliest processing start time, Respectively represent the current combination process O 1,j , O 2,j ,…,O n,j The minimum processing time is the same as the current combined process O 1,j -M1, O2,j -M1, ..., O n,j -Difference in processing time of M1, Respectively represent the current combination process O 1,j , O 2,j ,…,O n,j The minimum processing time is the same as the current combined process O 1,j -M k , O 2,j -M k ,…,O n,j -M k The difference in processing time;

[0043] For action space features: It represents the availability of the entire workpiece at the current scheduling time t, which is calculated by dividing the current available number of workpieces by the total number of workpieces. Indicates the availability of all machines at the current scheduling time, which is calculated by dividing the current number of available machines by the total number of machines. Indicates the average release time of the workpiece in the current state. The calculation method is the sum of the release time of all current workpieces divided by the total number of workpieces. Indicates the average release time of the machine in the current state. It is calculated by dividing the sum of the release times of all current machines by the total number of machines. It indicates the ratio of the optimal solution process that can be obtained in the current state, which is calculated by dividing the number of all the processes to be processed that can be processed on the machine with the minimum processing time by the total number of workpieces;

[0044] S2. Initial feature information of all workpieces in the current environment And the initial feature information of all machines in the current environment Input into the multi-head self-attention network, by inputting the original features of each workpiece or machine into multiple single-layer self-attention networks and splicing them together, we can obtain new features of each workpiece or machine such as Figure 5 As shown; at the same time, they are pooled separately to obtain pool feats.

[0045] S3: Pool feats obtained by pooling and feature information of action space in the current environment Splice and get the final information of the machine process pair, that is, the status Combine new workpiece features and new machine features with the initial feature information of the process-machine pair in the current environment Splice to get the final information of the action space, that is, the state

[0046] S4, the state Input action space agent, in the last layer of action space agent network, output probability distribution through SoftMax activation function, through the probability distribution of whether to transfer action space generated by action space agent, get action according to sampling or greedy strategy The status Enter the action agent, through the action probability distribution generated by the action agent, and obtain the action according to the sampling or greedy strategy The greater the probability of being selected, the more likely it is to be selected for actions with lower probabilities; while the greedy strategy always selects the action with the highest probability each time. The sampling strategy is used during the agent training process, and the greedy strategy is used when verifying the agent performance.

[0047] S5. According to the action Determine whether to transfer action space. If yes, Then the action space is transferred and reward is obtained If no, that is Then execute the action Get rewarded

[0048] The Reward is the difference between the next scheduled time and the current time, The Reward use Estimate a completion time Reuse Estimate the next completion time The final reward is

[0049] S6. Determine whether all scheduling has been completed. If yes, end the process; if not, return to step S1.

[0050] The agent training process in the present invention uses the idea of ​​reinforcement learning to continuously collect the environment (case state), the decisions made by the agent, the rewards obtained, and the next state to optimize the agent model. The specific steps are as follows:

[0051] Step a1: Initialize the parameters of the agent model, prepare the training set, test set, and a class for collecting decision-making process data;

[0052] Step a2: Sample an example from the training set, convert it into the initial environment for this training, clear the class that collects decision-making process data, and stop training if the maximum number of training times has been reached;

[0053] Step a3: Get the status according to the current environment Will Store the class that collects decision-making process data and stops the interaction between the agent and the environment if the current state is a final state;

[0054] Step a4: The agent determines the state Get Action Will Store in the class that collects data about the decision-making process;

[0055] Step a5: Determine the action Is it a conversion action space? If so, enter the next scheduling time t+1, return to step 3, and get a reward at the same time Will Store in the class that collects decision-making process data; if not, go to step 6;

[0056] Step a6: Execute the scheduling action Return to step b3 and get reward at the same time Will Store in the class that collects data about the decision-making process;

[0057] Step a7: Optimize the model based on the class of collected decision process data;

[0058] Step a8: This training is finished and returns to step a2;

[0059] For step a7 in the agent training process, the network optimization method adopted by the present invention is an improvement of the proximal policy optimization algorithm (PPO), which is based on the proximal policy optimization of clipping. The Actor-Critic framework is used in this training to efficiently optimize the model. The specific steps are as follows:

[0060] Step b1: The network is divided into the target network and the current network;

[0061] Step b2: Start the current training using the collected data. If the maximum number of training times has been reached, end the training;

[0062] Step b3: Using the collected data of the next state, use the Critic network of the target network to get the value of the next state;

[0063] Step b4: Use the value of the next state obtained by the Critic network of the target network and the reward obtained by taking action in the current state to calculate the TD (temporal difference) target;

[0064] Step b5: Use the current state and the Critic network of the current network to get the value v of the current state t ;

[0065] Step b6: Calculate the Critic network loss of the current network based on the value of the current state and the TD (temporal difference) target. The formula is: Where N is the number of times the agent needs to be scheduled for the final scheduling solution of an example, and t is each scheduling moment;

[0066] Among them, CriticLossCoef represents the loss coefficient of the Critic network, TDtarget t Represents the TD (temporal difference) target, and the formula is:

[0067] TD target t =r t +gamma*Critic old (S t+1 )

[0068] Among them, r t Reward, S t+1 represents the state, gamma represents the discount factor, and Critic old Represents the Critic network before soft update;

[0069] Step b7: Use the target network's critic network to get the value of the current state, and use it as the baseline to combine with the TD (temporal difference) target to calculate the TD error. The formula is TD delta t =TD target t -Critic old (S t ), where TD delta t represents TD error;

[0070] Step b8: Use the Actor network of the target network to obtain the value of the log function of the probability of executing the corresponding action in each state, recorded as old_log_p; use the Actor network of the current network to obtain the value of the log function of the probability of executing the corresponding action in each state, recorded as new_log_p;

[0071] Step b9: Calculate the value of new_log_p-old_log_p in the function f(x)=e x The value in is recorded as ratio; multiply ratio and TD error to get the result surr1; clip ratio according to the formula clip(ratio,1-ε,1+ε), and then multiply it with TD error to get the result surr2;

[0072] Step b10: According to surr1 and surr2, the loss of the Actor network of the current network is obtained. The formula is: ActorLossCoef represents the loss coefficient of the Actor network;

[0073] Step b11: Optimize the current network based on the loss of the Actor network and the Critic network of the current network, end this training, and return to step b2;

[0074] Step b12: soft update the target network according to the current network;

[0075] In order to fully demonstrate the effectiveness of the present invention, a large number of official data sets were used for testing. In detail, the data sets in the present invention include training data sets, verification data sets and test data sets, as shown in the following table:

[0076]

[0077] Hyperparameter settings of PPO algorithm

[0078] parameter value Number of training sessions 500，1000，1500 Policy update times 3 Verify instance interval number 5 Learning Rate <![CDATA[1×10 -3 <!-- 10 -->]]> Critic Loss Coef 0.5 Actor Loss Coefficient ActorLossCoef 1 Discount factor gamma 0.98 Clipping rate ε 0.2

[0079] The evaluation index of the present invention is the relative percentage deviation (RPD), also known as the optimality gap, and the calculation formula is as follows: Among them, C′ max represents the maximum completion time obtained using the method described, Refers to the maximum completion time of the best solution currently available for the problem;

[0080] In order to verify the effectiveness of the new method, this application uses the official dataset Dauzere to verify it. Compared with the existing leading network models (Lei et al. (Lei, et al., 2022); Song et al. (Song, Chen, Li, & Cao, 2023); Yuan et al. (Yuan, et al., 2024)) and priority scheduling rules (3 well-known composite PDRs: FIFO+EET, MWKR+EET and MOPNR+EET), on the public dataset Dauzere, the average optimal gap of FIFO+EET is 15.00%, the average optimal gap of MWKR+EET is 24.69%, the average optimal gap of MOPNR+EET is 32.17%, the average optimal gap of Song et al. is 8.88%, the average optimal gap of Lei et al. is 15.68%, the average optimal gap of Yuan et al. is 11.02%, and the average optimal gap of the present invention is 5.37%. The results show that the strategy model of the new method greatly reduces the average optimal Gap, which significantly confirms the effectiveness of the new technical method. The results are shown in the attached Figure 3 ;

[0081] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A flexible job shop scheduling method based on deep reinforcement learning and fusion of multi-head attention mechanism, characterized by: The following steps are involved: S1. According to the current flexible workshop environment, obtain the initial feature information of all workpieces in the current environment Initial feature information of all machines in the current environment Initial feature information of process-machine pairs in the current environment Feature information of the action space in the current environment And the characteristic information μt of the next scheduling time of the current environment; Among them, for the initial feature information of the workpiece: They represent the release time of workpiece J1 at the current scheduling time t, the release time of workpiece J2 at the current scheduling time t, ..., the release time of workpiece J at the current scheduling time t n The release time of J1, J2, ..., J n represents the operation of workpiece i, They represent the time difference of the release time of the same workpiece J1 at the current scheduling time t, the time difference of the release time of the same workpiece J2 at the current scheduling time t, ..., the time difference of the release time of the same workpiece J at the current scheduling time t n If the release time of the workpiece is less than or equal to the current scheduling time, the difference is 0. They represent the average processing time of the pending processing steps of workpiece J1 at the current scheduling time t, the average processing time of the pending processing steps of workpiece J2 at the current scheduling time t, ..., the average processing time of the pending processing steps of workpiece J at the current scheduling time t n The average processing time of the processing steps to be processed, They represent the minimum processing time of the pending processing procedure of workpiece J1 at the current scheduling time t, the minimum processing time of the pending processing procedure of workpiece J2 at the current scheduling time t, ..., the minimum processing time of the pending processing procedure of workpiece J at the current scheduling time t n The minimum processing time of the processing steps to be processed, They represent the remaining scheduled processes of workpiece J1 at the current scheduling time t, the remaining scheduled processes of workpiece J2 at the current scheduling time t, ..., the remaining scheduled processes of workpiece J at the current scheduling time t n The remaining processes to be scheduled; For the initial feature information of the machine: They represent the number of processes for processing and machine M1 at the current scheduling time t, the number of processes for processing and machine M2 at the current scheduling time t, and the number of processes for processing and machine M at the current scheduling time t. k The number of processes, They represent the release time of processing and machine M1 at the current scheduling time t, the release time of processing and machine M2 at the current scheduling time t, ..., the release time of processing and machine M at the current scheduling time t k The release time, They represent the difference between the release time of processing and machine M1 at the current scheduling time t, the difference between the release time of processing and machine M2 at the current scheduling time t, and the difference between the release time of processing and machine M at the current scheduling time t. k The difference in release time, if the mth machine M m If the release time of is less than or equal to the current scheduling time t, the difference is 0; For the initial feature information of the process-machine pair: They represent the process O at the current scheduling time t. 1，j , O 2，j ,…,O n，j Machining time on machine M1, They represent the process O at the current scheduling time t. 1，j , O 2，j ,…,O n，j On machine M k The processing time on They represent the process O at the current scheduling time t. 1，j , O 2，j ,…,O n，j The earliest start time of processing on machine M1, They represent the process O at the current scheduling time t. 1，j , O 2，j ,…,O n，j On machine M k The earliest processing start time, Respectively represent the current combination process O 1，j , O 2，j ,…,O n，j The minimum processing time is the same as the current combined process O 1，j -M1, O 2，j -M1, ..., O n，j -Difference in processing time of M1, Respectively represent the current combination process O 1，j , O 2，j ,…,O n，j The minimum processing time is the same as the current combined process O 1，j -M k , O 2，j -M k ,…,O n，j -M k The difference in processing time; For action space features: It represents the availability of the entire workpiece at the current scheduling time t, which is calculated by dividing the current available number of workpieces by the total number of workpieces. Indicates the availability of all machines at the current scheduling time, which is calculated by dividing the current number of available machines by the total number of machines. Indicates the average release time of the workpiece in the current state. The calculation method is the sum of the release time of all current workpieces divided by the total number of workpieces. Indicates the average release time of the machine in the current state. It is calculated by dividing the sum of the release times of all current machines by the total number of machines. It indicates the ratio of the optimal solution process that can be obtained in the current state, which is calculated by dividing the number of all the processes to be processed that can be processed on the machine with the minimum processing time by the total number of workpieces; S2. Initial feature information of all workpieces in the current environment And the initial feature information of all machines in the current environment Input the graph to the multi-head self-attention network to obtain new workpiece features and new machine features, and pool them separately to obtain poolfeats; S3: Poolfeats obtained by pooling and feature information of action space in the current environment and the next scheduling time characteristic μ t Splice and get the final information of the machine process pair, that is, the status Combine new workpiece features and new machine features with the initial feature information of the process-machine pair in the current environment Splice to get the final information of the action space, that is, the state S4, the state Input action space agent, through the probability distribution of whether to transfer action space generated by action space agent, get action according to sampling or greedy strategy The status Enter the action agent, through the action probability distribution generated by the action agent, and obtain the action according to the sampling or greedy strategy S5. According to the action Determine whether to perform action space transfer. If so, perform action space transfer and get reward If no, then perform the action Get rewarded S6. Determine whether all scheduling has been completed. If yes, end the process; if not, return to step S1.

2. The flexible job shop scheduling method based on deep reinforcement learning and fusion of multi-head attention mechanism according to claim 1 is characterized in that: The Reward is the difference between the next scheduling time and the current time, and its specific expression is as follows: Among them, Time t Indicates the current time, Time t+1 Indicates the next scheduling time.

3. The flexible job shop scheduling method based on deep reinforcement learning and fusion of multi-head attention mechanism according to claim 1 is characterized in that: The Reward use Estimate a completion time Reuse Estimate the next completion time The expression for the final reward is as follows: Among them, C max (S t ) represents the completion time, C max (S t+1 ) indicates the next completion time.

Images