Flexible job shop scheduling method based on deep reinforcement learning

Abstract

The invention discloses a flexible job shop scheduling method based on deep reinforcement learning, and belongs to the field of intelligent manufacturing and intelligent scheduling, and the method comprises the steps: firstly, taking the minimization of the maximum job completion time and the maximization of the average machine utilization rate as optimization targets, and obtaining a target model; then representing a flexible job shop scheduling problem by using a disjunction graph model to obtain a scheduling state; according to the scheduling state and the target model, an end-to-end learning framework is constructed, and the framework is combined with a multi-attention network for deep feature extraction and a decision network for extensible decision, and a Markov decision process is utilized to obtain a job scheduling model; and finally, the multiple attention network constructs a production adaptability operation-AGV-machine scheme by using the complex relationship among the operation, the AGV and the machine to support the decision of the decision network, and obtains a scheduling schedule by using the job scheduling model to complete flexible job shop scheduling. According to the method, complex modes and relations in data can be better captured, and the performance and generalization ability of the model are improved.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing and intelligent scheduling, and in particular to a flexible job shop scheduling method based on deep reinforcement learning. Background Art

[0002] In today's increasingly competitive global economy, the manufacturing industry is undergoing profound changes. With the rapid development of information technology, especially the continuous emergence of cutting-edge technologies such as big data, artificial intelligence, and the Internet of Things, industrial intelligent manufacturing has become an important direction for the transformation and upgrading of the manufacturing industry. In the past, traditional manufacturing mainly relied on manual operation and experience-based decision-making, with low production efficiency, unstable product quality, and difficulty in meeting the rapidly changing needs of the market. Industrial intelligent manufacturing has achieved automation, digitization, and intelligence of the production process by deeply integrating advanced information technology with the manufacturing industry.

[0003] The Chinese government has introduced a series of policies to encourage and support the development of industrial intelligence. In the system of industrial intelligent manufacturing, intelligent scheduling plays a vital role. Effective intelligent scheduling can optimize the allocation of production resources, improve equipment utilization, reduce production cycles, and reduce production costs, thereby enhancing the competitiveness of enterprises. However, my country's manufacturing industry still faces many challenges in intelligent scheduling.

[0004] Flexible job shop scheduling is an important research area in modern production management. It aims to optimize resource allocation and job sequence in the production process to improve production efficiency, reduce costs and meet customer needs. In traditional job shop scheduling, the correspondence between machines and processes is usually fixed. However, flexible job shop scheduling introduces more flexibility. Machines are no longer strictly limited to specific processes, but can handle a variety of different processes, which greatly increases the possibility and complexity of scheduling. The goals of flexible job shop scheduling usually include minimizing production cycle, minimizing production costs, maximizing equipment utilization, etc. In order to achieve these goals, it is necessary to comprehensively consider multiple factors, such as the processing order of workpieces, the available time of machines, the processing time of different processes, transportation time, etc.

[0005] In recent years, the research and application of deep reinforcement learning in the flexible job shop scheduling problem has made some progress. For the flexible job shop scheduling problem, the deep reinforcement learning method trains the intelligent scheduling agent to enable it to observe the state from the environment and make decisions. The core idea of ​​this method is to optimize the scheduling strategy through interaction with the environment to minimize the maximum completion time. However, the existing deep reinforcement learning methods have certain drawbacks in capturing the characteristics of operation-AGV-machine, and there is an urgent need to adopt a mechanism to solve this problem.

[0006] At this stage, it is very important to improve the ability to handle important features and relationships between machines and operations. More efficient computing methods and more accurate feature extraction methods can enable the network to better cope with large-scale and high-dimensional data. The optimized network may have improved performance, better handle time series, and be more efficient in the use of computing resources. Reducing training time and computing costs allows effective model training and deployment even with limited resources. At the same time, enhancing the generalization ability of the network so that it not only performs well on specific training data sets, but also maintains good performance on new and unseen data, thereby increasing the applicability and practicality of the network. Therefore, it is of great significance to study new methods for flexible job shop scheduling based on deep reinforcement learning. Summary of the invention

[0007] The purpose of the present invention is to provide a flexible job shop scheduling method based on deep reinforcement learning, which can utilize its specific architecture and training method to better capture complex patterns and relationships in the data, thereby improving the performance and generalization ability of the model.

[0008] To achieve the above object, the technical solution adopted by the present invention is:

[0009] A flexible job shop scheduling method based on deep reinforcement learning includes the following steps:

[0010] S1, taking minimization of the maximum completion time of the job and maximization of the average machine utilization as the optimization objectives, the target model is obtained;

[0011] S2. Use the disjunctive graph model to represent the flexible job shop scheduling problem and obtain the scheduling status;

[0012] S3. According to the scheduling state and the target model, an end-to-end learning framework is constructed. The framework combines a multiple attention network for deep feature extraction and a decision network for scalable decision making, and uses a Markov decision process to obtain a job scheduling model; the multiple attention network includes an AGV information attention block, an operation information attention block, and a machine information attention block that are interconnected, and the decision network is trained in a deep reinforcement learning manner;

[0013] S4. The multi-attention network uses the complex relationship between operations, AGVs, and machines to construct a production-adaptive operation-AGV-machine solution to support the decision-making of the decision network, and uses the job scheduling model to obtain a scheduling schedule to complete flexible job shop scheduling.

[0014] The beneficial effects of the present invention are as follows: the method makes optimal decisions in flexible job shop scheduling based on deep reinforcement learning and graph attention networks; the method can flexibly select optimization targets according to demand, and achieve real-time optimization of targets through collaboration between intelligent agents, while meeting market demand, it can also further improve production efficiency and solve large-scale shop scheduling tasks.

[0015] Furthermore, the expression of the target model in step S1 is: Where FJSPMOD is the target model; Minimize is the minimization function; Makespan is the maximum completion time of the i-th job to complete all operations; max is the maximization function; The time it takes for the i-th job to complete all operations; i is the job number; n is the number of jobs; n i is the total number of operations of the ith job; U ave is the average machine utilization rate; m is the number of machines; k is the machine number; j is the operation number of the i-th job; t ijk is the processing time of the jth operation of the i-th job on the k-th machine; X ijk It is an identifier of whether the jth operation of the ith job can be performed on the kth machine. If yes, it is 1, otherwise, it is 0; time is the current time of the environment.

[0016] The beneficial effects of the above scheme are as follows: the flexible job shop scheduling problem is specifically described through mathematical modeling to optimize the job scheduling scheme and maximize production efficiency.

[0017] Furthermore, the expression of the scheduling state in step S2 is: V=(O,M,C,D,G) In the formula, V is the scheduling state, O is the operation node, M is the machine node, C is the conjunction arc, D is the disjunction arc, G is the AGV node, and O is the ij is the jth operation of the ith job; i is the job number; j is the operation number of the ith job; Start is the virtual start node with a processing time of zero; End is the end node.

[0018] The above scheme has the following beneficial effects: the scheduling state is represented by a disjunctive graph, which clearly describes the complex relationship between each operation-AGV-machine in the job shop. Through the connection of nodes and edges, the order of different operations, the compatibility of operations-machines, and the availability of AGVs are intuitively presented. This graphical modeling makes the problem easier to understand and helps with the formal description and visual analysis of the problem.

[0019] Furthermore, the step S3 is specifically as follows:

[0020] S301, obtaining instance data;

[0021] S302. According to the instance data, scheduling status and target model, the intelligent agent is trained using the Markov decision process, and multiple attention networks are used for operation, AGV and machine feature extraction. The policy distribution and state value are output through the decision network, and the proximal policy optimization algorithm is used to train the intelligent agent. The state, action and reward value are selected and recorded to obtain the job scheduling model.

[0022] The beneficial effects of the above scheme are as follows: this method can deeply extract the features of the operation-AGV-machine combination pair, comprehensively consider the advantages of objectives, learning and strong adaptability, and strategic process optimization, so that the model can effectively adapt to more complex job scheduling problems, improve the generalization ability and robustness of the model, and achieve more flexible production scheduling.

[0023] Furthermore, the step S302 is specifically as follows:

[0024] S3021. Randomly generate training instances according to preset rules, including operation length, operation processing time, and AGV transportation time;

[0025] S3022, using the scheduling state, input the operation feature into the operation information attention block for calculation, and output the operation node feature; input the AGV feature into the AGV information attention block for calculation, and output the AGV node feature; input the machine feature into the machine information attention block for calculation, and output the machine node feature, respectively average pooling and connecting the operation node feature, AGV node feature and machine node feature to generate the operation global feature, AGV global feature and machine global feature;

[0026] S3023, inputting the operation global features, AGV global features and machine global features into the strategy value network respectively, and outputting the scheduling strategy and state value; the decision network includes a plurality of parallel strategy value networks, and the strategy value network includes a strategy network for generating strategies and a value network for evaluating the value of the current state;

[0027] S3024. Use the scheduling status and policy distribution to select the operation-AGV-machine pair, update the operation start time and end time, AGV empty transport time, load transport start time and load transport end time in the operation-AGV-machine pair, use the proximal policy optimization algorithm to train the intelligent agent, calculate the reward according to the target to be optimized, and update the job scheduling model parameters until the model converges.

[0028] Furthermore, the total loss function in the proximal strategy optimization algorithm is: L value (φ)=(V φ (s t )-V t target ) 2 V t target =R t +γV φ (s t+1 ) L entropy (π)=-∑ a π(a|s)logπ(a|s) L PPO (θ,φ)=-L clip (θ)+c 1 L value (φ)-c 2 L entropy (π) Where, L clip (θ) is the loss function of the policy network; θ is the parameter of the policy network; min{·} is the minimization function; r t (θ) is the ratio of the new and old strategies; is the advantage function at time t; t is the time; clip(·) is the clipping function; ε is the clipping parameter; π(·) is the new strategy; a t is the action of the agent at time t; θ is the network parameter corresponding to the new strategy; π old (·) is the old strategy; θ old is the network parameter corresponding to the old strategy; L value (φ) is the loss function of the value network; φ is the parameter of the value network; V φ (·) is the state value of the value network; s t is the state of the agent at time t; V t traget is the target value at time t, which is the weighted sum of the immediate reward and the future value state; R t is the immediate reward, which indicates the benefit of the current action; γV φ (s t+1 ) represents the possible future return; γ is the discount factor; s t+1 is the new state of the agent; L entropy is the entropy regularization term, which is used to increase the randomness of the strategy and prevent the strategy from converging prematurely; π is the discrete probability distribution of the input, that is, the probability distribution of selecting action a in state s; L PPO (θ, φ) is the total loss function; c1 and c 2 is a weight hyperparameter; Q(·) is the state-action value function; V(·) is the state value function.

[0029] The beneficial effects of the above scheme are: improving the exploratory nature of the strategy, facilitating a balance between exploration and development, preventing model overfitting, and facilitating the stability and efficiency of the algorithm.

[0030] Furthermore, the step S3022 is specifically as follows:

[0031] Using the scheduling state, the operation features and operation masks are input into the operation information attention block. After linear change and rolling combination operations, the domain information of the source node operation features and the domain information of the target node operation features are generated. The source node operation features and the target node operation features are broadcast and added to generate interaction features. After that, the interaction features are nonlinearly transformed, masked, and softmax normalized. The neighborhood features of the operation nodes are weighted and aggregated using the attention weights to generate new operation features.

[0032] The operation feature, operation mask, workpiece candidate value, machine feature, machine mask and machine competition index are input into the machine information attention block; through the filter connected between the operation information attention block and the machine information attention block, the machine competition index in the environment is matrix-multiplied with the operation feature matrix to obtain the machine competition variable; then the machine feature and the edge feature between the operation and the machine are linearly transformed, the source node machine feature, the target node machine feature and the edge node feature between the operation and the machine are broadcast and added, and then the attention coefficient between the two machines is nonlinearly transformed, masked and softmax normalized, and the machine nodes with competitive relationship are weightedly aggregated using the attention weight to generate a new machine feature;

[0033] The operation features, operation masks, workpiece candidate values, AGV features, AGV masks and AGV competition indexes are passed into the AGV information attention block; through the filter connected between the operation information attention block and the AGV information attention block, the AGV competition index in the environment is matrix-multiplied with the operation feature matrix to obtain the AGV competition variable, and then the AGV features and the edge features between the operation and the AGV are linearly transformed, the source node AGV features, the target node AGV features and the edge node features between the operation and the AGV are broadcast and added, and then the attention coefficient between the two AGVs is nonlinearly transformed, masked, and softmax normalized, and the attention weights are used to perform weighted aggregation on the AGV nodes with competing relationships to generate new AGV features.

[0034] The beneficial effects of the above scheme are: deep extraction of operation-AGV-machine features, ensuring the accuracy of input features of the optimization strategy network, and improving the reliability and stability of the algorithm.

[0035] Furthermore, the S3022 specifically includes: there are multiple operation information attention blocks, AGV information attention blocks and machine information attention blocks, which respectively perform feature calculations on their respective multiple attention heads, generate outputs for each attention head, and then merge the multiple attention head outputs using a splicing mode or an averaging mode.

[0036] Furthermore, the step S3023 is specifically as follows:

[0037] S30231. Using the policy network, the extracted operation features, AGV features, machine features, operation-AGV pair features, machine-AGV pair features, and global features are input to generate a scalar for each candidate action, and the operation-AGV-machine pair is obtained. The obtained scalar is calculated as follows: In the formula, μ(a t |s t ) is the priority index of action a selected in state s at time t; MLP θ is a multilayer perceptron based on θ; θ is the network parameter of MLP; The output features of the operation information attention block; is the output feature of the AGV information attention block; is the output feature of the machine information attention block; It is the global feature of the FJSP instance; is the output feature of the operator-AGV pair; is the output feature of the operation-machine pair;

[0038] S30232. Use the softmax function to normalize the priority index scalar and output the probability distribution of the selected action. The calculation formula is: In the formula, π θ (a t |s t ) is the probability of selecting the operation-AGV-machine pair; a t is the action of the agent; s t is the state at time t; exp(·) is an exponential function with a natural constant as the base; μ(a t |s t ) is the priority index of action a selected in state s at time t; b t is the action element in the set of optional actions for the agent; b tis the set of optional actions at time t; θ is the network parameter of MLP.

[0039] The beneficial effects of the above scheme are: operation-AGV-machine selection is performed so that each operation can be processed on a compatible machine, and operations that need to be transported can be transported to the corresponding machine, further optimizing important indicators such as maximum completion time and machine utilization.

[0040] Further, step S3024 is specifically as follows:

[0041] S30241. Update candidate operations according to the operation, AGV and machine index of the selected action. If the current operation is not the last operation of the job, update the candidate operation. Otherwise, mark it as completed.

[0042] S30242, using the scheduling status, update the AGV transportation time, take the maximum value of the operation and the current idle time of the AGV to update the AGV empty transportation start time, update the AGV load transportation time according to the AGV empty transportation time, take the AGV load transportation end time and the maximum idle time of the machine to update the operation start time, update the operation completion time according to the operation processing time, update the above time information, and obtain the scheduling time schedule;

[0043] S30243. Update the scheduled maximum completion time according to the operation completion time;

[0044] S30244. Update the operation status according to the index value of the completed operation, mark the scheduled operation and adjust the remaining operation-AGV-machine relationship matrix, delete the completed operation, update the operation mask, AGV mask, machine mask, operation feature, machine feature, operation-AGV pair feature, AGV-machine pair feature.

[0045] The beneficial effects of the above scheme are: by updating state features through action selection, calculating candidate value features, and updating masks, the scheduling schedule can be calculated and the maximum completion time can be shortened as much as possible while improving machine utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is the overall flow chart of the present invention;

[0047] Figure 2 It is the technical roadmap of the present invention;

[0048] Figure 3 This is a flow chart of the flexible job shop scheduling framework based on graph neural network and deep reinforcement learning of the present invention;

[0049] Figure 4 is a disjunctive graph representing a scheduling problem in an embodiment of the present invention;

[0050] Figure 5 A multi-attention network diagram for extracting operation-AGV-machine features in an embodiment of the present invention;

[0051] Figure 6 This is the actor network framework diagram of the present invention;

[0052] Figure 7 It is the FJSP-AGV scheduling scheme diagram of the present invention;

[0053] Figure 8 The following is a diagram showing the solution in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0055] A flexible job shop scheduling method based on deep reinforcement learning includes the following steps:

[0056] S1, taking minimization of the maximum completion time of the job and maximization of the average machine utilization as the optimization objectives, the target model is obtained;

[0057] S2. Use the disjunctive graph model to represent the flexible job shop scheduling problem and obtain the scheduling status;

[0058] S3. According to the scheduling state and the target model, an end-to-end learning framework is constructed. The framework combines a multiple attention network for deep feature extraction and a decision network for scalable decision making, and uses a Markov decision process to obtain a job scheduling model; the multiple attention network includes an AGV information attention block, an operation information attention block, and a machine information attention block that are interconnected, and the decision network is trained in a deep reinforcement learning manner;

[0059] S4. The multi-attention network uses the complex relationship between operation-AGV-machine to construct a production adaptive operation-AGV-machine solution to support the decision-making of the decision network, and uses the job scheduling model to obtain the scheduling schedule to complete the flexible job shop scheduling.

[0060] Furthermore, the expression of the target model in step S1 is: Where FJSPMOD is the target model; Minimize is the minimization function; Makespan is the maximum completion time of the i-th job to complete all operations; max is the maximization function; The time it takes for the i-th job to complete all operations; i is the job number; n is the number of jobs; n i is the total number of operations of the ith job; U aveis the average machine utilization rate; m is the number of machines; k is the machine number; j is the operation number of the i-th job; t ijk is the processing time of the jth operation of the i-th job on the k-th machine; X ijk It is an identifier of whether the jth operation of the ith job can be performed on the kth machine. If yes, it is 1, otherwise, it is 0; time is the current time of the environment.

[0061] Furthermore, the expression of the scheduling state in step S2 is: V=(O,M,C,D,G) In the formula, V is the scheduling state, O is the operation node, M is the machine node, C is the conjunction arc, D is the disjunction arc, G is the AGV node, and O is the ij is the jth operation of the ith job; i is the job number; j is the operation number of the ith job; Start is the virtual start node with a processing time of zero; End is the end node.

[0062] Furthermore, the step S3 is specifically as follows:

[0063] S301, obtaining instance data;

[0064] S302. Based on instance data, scheduling status and target model, the intelligent agent is trained using the Markov decision process, and multiple attention networks are used for operation-AGV-machine feature extraction to help the intelligent agent understand the complex relationship between jobs. The action strategy distribution and state value are output through the decision network using the 'strategy-value' method. The intelligent agent is trained using the proximal policy optimization algorithm, and the state, action and reward values ​​are selected and recorded to obtain the job scheduling model.

[0065] Furthermore, the step S302 is specifically as follows:

[0066] S3021. Randomly generate training instances according to preset rules, including operation length, operation processing time, and AGV transportation time;

[0067] S3022. Using the scheduling state, input the operation feature into the operation information attention block for calculation, and output the operation node feature; input the AGV feature into the AGV information attention block for calculation, and output the AGV node feature; input the machine feature into the machine information attention block for calculation, and output the machine node feature; respectively average pool the operation node feature, AGV node feature, and machine node feature and connect them to generate the operation global feature, AGV global feature, and machine global feature;

[0068] S3023. Input the global features of the operation, the global features of the AGV and the global features of the machine into the policy value network respectively, and output the scheduling strategy and the state value; each scheduling strategy constitutes a policy distribution. The decision network includes a plurality of parallel policy value networks, and the policy value network includes a policy network for generating a strategy and a value network for evaluating the value of the current state; the policy network and the value network use two multi-layer perceptrons respectively, aiming to generate random strategies and state value estimates.

[0069] S3024. Use the scheduling status and policy distribution to select the operation-AGV-machine pair, update the operation start time and end time, AGV empty transport time, load transport start time and load transport end time in the operation-AGV-machine pair, use the proximal policy optimization algorithm to train the intelligent agent, calculate the reward according to the target to be optimized, and update the job scheduling model parameters; until the model converges.

[0070] The total loss function in the proximal strategy optimization algorithm consists of the following three parts: L value (φ)=(V φ (s t )-V t target ) 2 V t target =R t +γV φ (s t+1 ) L entropy (π)=-∑ a π(a|s)logπ(a|s) L PPO (θ,φ)=-L clip (θ)+c 1 L value (φ)-c 2 L entropy (π) Where, L clip (θ) is the loss function of the policy network; θ is the parameter of the policy network; min{·} is the minimization function; r t (θ) is the ratio of the new and old strategies; is the advantage function at time t; t is the time; clip(·) is the clipping function; ε is the clipping parameter; π(·) is the new strategy; a t is the action of the agent at time t; θ is the network parameter corresponding to the new strategy; π old (·) is the old strategy; θold is the network parameter corresponding to the old strategy; L value (φ) is the loss function of the value network; φ is the parameter of the value network; V φ (·) is the state value of the value network; s t is the state of the agent at time t; V t traget is the target value at time t, which is the weighted sum of the immediate reward and the future value state; R t is the immediate reward, which indicates the benefit of the current action; γV φ (s t+1 ) represents the possible future return; γ is the discount factor; s t+1 is the new state of the agent; L entropy is the entropy regularization term, which is used to increase the randomness of the strategy, encourage exploration, and prevent the strategy from converging prematurely; π is the discrete probability distribution of the input, that is, the probability distribution of selecting action a in state s; L PPO (θ, φ) is the weighted sum of the three parts; c 1 and c 2 is a weight hyperparameter used to balance the three parts; Q(·) is the state-action value function; V(·) is the state value function;

[0071] S3025, judging whether all operations to be processed in the sampled data have been scheduled and transported to the corresponding machines, if so, directly proceeding to step S3026, otherwise, returning to step S3022;

[0072] S3026: Determine whether the first formula is valid. If so, verify the job scheduling model, obtain the verification result, and proceed to step S3027. Otherwise, directly proceed to step S3027. The first formula is: imodN val =0 In the formula, i is the number of iterations; mod is the remainder; N val To verify the threshold;

[0073] S3027, determine whether the second formula is valid, if so, return to step S3021, otherwise, directly enter S3028; the expression of the second formula is: imodN r = = 0 Where N r is the sampling threshold;

[0074] S3028, determine whether the third formula is valid, if so, directly go to step S2029, otherwise, return to step S3022; the expression of the third formula is: i=N ep Where N ep is the maximum iteration threshold;

[0075] S3029. Set an observation interval and determine whether the strategy verification result converges within the observation interval. If so, end the training and obtain the job scheduling model. Otherwise, adjust the parameters of the agent and return to step S3022.

[0076] Further: the step S3022 is specifically as follows:

[0077] S30221. Using the scheduling state, input the operation feature, machine feature, AGV feature, operation mask, machine mask, AGV mask, workpiece candidate value, machine competition index, and AGV competition index into the multi-attention network model, and process them in parallel through the operation information attention block, the AGV information attention block, and the machine information attention block;

[0078] S30222, input the operation feature and the operation mask into the operation information attention block, perform linear transformation and rolling combination operations on the input operation feature vector, splice the current feature with the predecessor node feature and the successor node feature in sequence, and generate domain information of the source node operation feature;

[0079] S30223, dividing the weight matrix of the operation information attention block into two parts, performing matrix multiplication on the source node operation feature and the target node operation feature respectively, and generating two feature projections;

[0080] S30224, performing a rolling combination operation on the target node vector, splicing the current feature with the predecessor node feature and the successor node feature in sequence, and generating domain information of the target node operation feature;

[0081] S30225. Broadcast and add the source node operation features and the target node operation features to generate interaction features to represent the interaction relationship between the source node and the target node; and use the LeakyReLU activation function to perform nonlinear transformation on the interaction features to enhance the nonlinear ability of feature representation. The attention coefficient calculation formula for the interaction features between the source node and the target node is: In the formula, e i,j,p Operation node O ij With operation node O ip The attention coefficient of the interaction features between them; LeakyReLU(·) is the activation function; is the weight matrix vector; T represents the transposition of the weight matrix; W is the input feature tensor; For operation O ij The input features of For operation O ipThe input features of , [(·)∥(·)] is a cascade operation;

[0082] S30226. Select a negative tensor or an attention coefficient according to the value of the input operation mask, and use the negative tensor to mask the invalid position so that it is not affected in the subsequent softmax;

[0083] S30227, for all operation masks, e i,j,p , use the softmax function for normalization and convert the interaction score into attention weight. The calculation formula of the softmax function is: In the formula, x i is the i-th score of the input vector, exp(·) is the exponential function; It is the sum of all elements in the input vector;

[0084] S30228. Apply regularization techniques to the normalized attention weights to randomly reset some weights to zero, thereby reducing the risk of overfitting the model and preventing the attention mechanism from being overly focused on certain specific locations.

[0085] S30229. Use the attention weight to perform weighted aggregation on the neighborhood features of the operation node to generate a new operation feature representation. The calculation formula of the new feature is: In the formula, For operation O ij The new features of , σ(·) is a nonlinear activation function; Indicates the operation node O ij Calculate and sum with its predecessor node and successor node; α i,j,p represents the probability distribution obtained by normalizing the attention coefficient; W is the input feature tensor; For operation O ip Input features of

[0086] S302210, inputting the operation features and the operation masks into multiple operation information attention blocks, and performing feature calculations on multiple attention heads to generate outputs of each attention head;

[0087] S302211. Use the concatenation mode or the average mode to merge the outputs of multiple attention heads. After concatenation, the output features of each attention head are concatenated by dimension. This mode retains the independent features of each attention head and is suitable for scenarios where multi-head information needs to be retained. That is, the features of each node are composed of the outputs of all attention heads concatenated by dimension. In the average mode, the output of each attention head is no longer directly concatenated, but a new dimension is added to store the features of multiple heads, and the average value is calculated to average the output features of each head to generate a fused representation, that is, the feature of each node is the average of the output features of all attention heads.

[0088] S302212. Using the scheduling state, input the operation feature, the operation mask, the workpiece candidate value, the machine feature, the machine mask, and the machine competition index into the machine information attention block;

[0089] S302213, through the filter connected between the operation information attention block and the machine information attention block, use the machine competition index in the environment to perform matrix multiplication with the operation feature matrix to obtain a machine competition variable, which is used to represent the competition intensity of two machines competing for a certain operation;

[0090] S302214. Input the machine features, machine masks, and machine competition variables into the machine information attention block, perform linear transformation on the machine features, and perform matrix multiplication on the machine competition variables and the edge feature matrix between the operation and the machine, which is equivalent to performing linear transformation on the edge features between the operation and the machine. The calculation formula is: h' M =Z M h M h′ edge =Z edge h eage In the formula, h' M Represents the characteristics after linear transformation of machine nodes; Z M is the weight matrix of the machine node; h M is the original input feature of the machine node; h e ' dge Represents the characteristics of the operation-machine edge node after linear transformation; Z edge is the weight matrix of the operation-machine edge node; h edge is the original input edge feature of the machine node;

[0091] S302215. Divide the weight matrix of the machine information attention block into two parts, perform matrix multiplication on the source node machine feature and the target node machine feature respectively, and generate two feature projections; perform linear transformation on the edge feature between the operation and the machine, broadcast and add the obtained source node machine feature, target node machine feature and edge node feature between the operation and the machine, and apply the LeakyReLU activation function to perform nonlinear transformation on the obtained attention coefficient between the two machines to generate a new tensor; the calculation formula of the attention coefficient between the two machines is expressed as follows: In the formula, u kq Indicates machine M k and Machine M q The attention coefficient between them; LeakyReLU(·) is the activation function; is the weight matrix vector; T represents the transposition of the weight matrix; Z 1 The weight matrix obtained by matrix multiplication between machines; Z 2 The weight matrix obtained by matrix multiplication for the machine and operation; For machine M k The original input features of For machine M q The original input features of kq Compete for machine variables;

[0092] S302216. Select a negative tensor or attention coefficient according to the value of the input machine mask, and use the negative tensor to mask the invalid position so that it is not affected in the subsequent softmax;

[0093] S302217, for all masked u kq , use the softmax function for normalization and convert the interaction score into attention weight. The calculation formula of the softmax function is: In the formula, x i is the i-th score of the input vector, exp(·) is the exponential function; It is the sum of all elements in the input vector;

[0094] S302218. Apply regularization techniques to the normalized attention weights to randomly reset some weights to zero, thereby reducing the risk of overfitting the model and preventing the attention mechanism from being overly focused on certain specific locations.

[0095] S302219. Use the attention weight to perform weighted aggregation on the competing machine nodes to generate a new machine feature representation. The calculation formula for the new feature is: In the formula, For machine M k and Machine M q The new features of , σ(·) is a nonlinear activation function; ∑ k Represents the machine node M k Perform sum operation; β kq represents the probability distribution obtained by normalizing the attention coefficient; Z is the input feature tensor; For machine M k and Machine M q Input features of

[0096] S302220, inputting the operation features, operation masks, workpiece candidate values, machine features, machine masks, and machine competition indexes into multiple machine information attention blocks, and performing feature calculations on multiple attention heads to generate outputs of each attention head;

[0097] S302221. Use the concatenation mode or the average mode to merge the outputs of multiple attention heads. After concatenation, the output features of each attention head are concatenated by dimension. This mode retains the independent features of each attention head and is suitable for scenarios where multi-head information needs to be retained. That is, the features of each node are composed of the outputs of all attention heads concatenated by dimension. In the average mode, the output of each attention head is no longer directly concatenated, but a new dimension is added to store the features of multiple attention heads, and the average value is calculated to average the output features of each attention head to generate a fused representation, that is, the feature of each node is the average of the output features of all attention heads.

[0098] S302222. Using the scheduling state, input the operation feature, the operation mask, the workpiece candidate value, the AGV feature, the AGV mask, and the AGV competition index into the AGV information attention block;

[0099] S302223, through the filter connected between the operation information attention block and the AGV information attention block, the AGV competition index in the environment is matrix-multiplied with the operation feature matrix to obtain an AGV competition variable, which is used to represent the competition intensity of two AGVs competing for a certain operation;

[0100] S302224. Input the AGV features, AGV masks, and AGV competition variables into the AGV information attention block, perform linear transformation on the AGV features, and perform matrix multiplication on the AGV competition variables and the edge feature matrix between the operation and the AGV, which is equivalent to performing a linear transformation on the edge feature between the operation and the AGV. The calculation formula is: h' A =Z A h A h' aedge =Z aedge h aeage In the formula, h' A Represents the features after linear transformation of AGV nodes; Z A is the weight matrix of the AGV node; h A is the original input feature of the AGV node; h' aedge Represents the characteristics of the operation-AGV edge node after linear transformation; Z aedge is the weight matrix of the operation-AGV edge node; h aedge is the original input edge feature of the AGV node;

[0101] S302225. Divide the weight matrix of the AGV information attention block into two parts, perform matrix multiplication on the source node AGV feature and the target node AGV feature respectively, and generate two feature projections; perform linear transformation on the edge features between the AGV and the operation, broadcast and add the obtained source node AGV features, target node AGV features and edge node features between the operation and the AGV, and apply the LeakyReLU activation function to perform nonlinear transformation on the obtained attention coefficient between the two AGVs to generate a new tensor; the calculation formula of the attention coefficient between the two AGVs is expressed as follows: In the formula, v rs AGVA r and AGVA s The attention coefficient between them; LeakyReLU(·) is the activation function; is the weight matrix vector; T represents the transposition of the weight matrix; V 1 V is the weight matrix obtained by matrix multiplication between AGV and AGV; 2 The weight matrix obtained by matrix multiplication of AGV and operation; For AGVA r The original input features of For AGVA s The original input features of rs is the AGV competition variable;

[0102] S302226. Select a negative tensor or an attention coefficient according to the value of the input AGV mask, and use the negative tensor to mask the invalid position so that it is not affected in the subsequent softmax;

[0103] S302227, for all masked v rs, use the softmax function for normalization and convert the interaction score into attention weight. The calculation formula of the softmax function is: In the formula, x i is the i-th score of the input vector, exp(·) is the exponential function; It is the sum of all elements in the input vector;

[0104] S302228. Apply regularization techniques to the normalized attention weights to randomly reset some weights to zero, thereby reducing the risk of overfitting the model and preventing the attention mechanism from being overly focused on certain specific locations.

[0105] S302229. Use the attention weight to perform weighted aggregation on the competing AGV nodes to generate a new AGV feature representation; the calculation formula for the new feature is: In the formula, For AGVA r and AGVA s The new features of , σ(·) is a nonlinear activation function; ∑ a Indicates that for AGV node A a Perform sum operation; η rs represents the probability distribution obtained by normalizing the attention coefficient; V is the input feature tensor; For AGVA r and AGVA s Input features of

[0106] S302230, inputting the operation features, operation masks, workpiece candidate values, AGV features, AGV masks, and AGV competition indexes into multiple AGV information attention blocks, and performing feature calculations on multiple attention heads to generate outputs of each attention head;

[0107] S302231. Use the concatenation mode or the average mode to merge the outputs of multiple attention heads. After concatenation, the output features of each attention head are concatenated by dimension. This mode retains the independent features of each attention head and is suitable for scenarios where multi-head information needs to be retained. That is, the features of each node are composed of the outputs of all attention heads concatenated by dimension. In the average mode, the output of each attention head is no longer directly concatenated, but a new dimension is added to store the features of multiple attention heads, and the average value is calculated to average the output features of each attention head to generate a fused representation, that is, the feature of each node is the average of the output features of all attention heads.

[0108] S302232. Use the new features output by the operation information attention block, the AGV information attention block, and the machine information attention block, eliminate the influence of invalid nodes, use average pooling to obtain the global features, and connect the three global features to form the global features of the FJSP instance; the calculation formula is as follows: In the formula, is the global feature of the FJSP instance; L represents the number of multi-head attention heads; |O| is the number of operation nodes; is the output feature of the operation information attention block; |A| is the number of AGV nodes; is the output feature of the AGV information attention block; |M| is the number of machine nodes; is the output feature of the machine information attention block.

[0109] Furthermore, the step S3023 is specifically as follows:

[0110] S30231. Using the policy network, the extracted operation features, AGV features, machine features, operation-AGV pair features, machine-AGV pair features, and global features are input to generate a scalar for each candidate action, and the operation-AGV-machine pair is obtained. The obtained scalar is calculated as follows: In the formula, μ(a t |s t ) is the priority index of action a selected in state s at time t; MLP θ is a multilayer perceptron based on θ; θ is the network parameter of MLP; The output features of the operation information attention block; is the output feature of the AGV information attention block; is the output feature of the machine information attention block; It is the global feature of the FJSP instance; is the output feature of the operator-AGV pair; is the output feature of the operation-machine pair;

[0111] S30232. Use the softmax function to normalize the priority index scalar and output the probability distribution of the selected action. The calculation formula is: In the formula, π θ (a t |s t ) is the probability of selecting the operation-AGV-machine pair; a t is the action of the agent; s tis the state at time t; exp(·) is an exponential function with a natural constant as the base; μ(a t |s t ) is the priority index of action a selected in state s at time t; b t is the action element in the set of optional actions for the agent; b t is the set of optional actions at time t; θ is the network parameter of MLP;

[0112] Furthermore, the step S3024 is specifically as follows:

[0113] S30241. Update candidate operations according to the operation, AGV and machine index of the selected action. If the current operation is not the last operation of the job, update the candidate operation. Otherwise, mark it as completed.

[0114] S30242, using the scheduling status, update the AGV transportation time, take the maximum value of the operation and the current idle time of the AGV to update the AGV empty transportation start time, update the AGV load transportation time according to the AGV empty transportation time, take the AGV load transportation end time and the maximum idle time of the machine to update the operation start time, update the operation completion time according to the operation processing time, update the above time information, and obtain the scheduling time schedule;

[0115] S30243. Update the scheduled maximum completion time according to the operation completion time;

[0116] S30244. Update the operation status according to the index value of the completed operation, mark the scheduled operation and adjust the remaining operation-AGV-machine relationship matrix, delete the completed operation, update the operation mask, AGV mask, machine mask, update the operation feature, update the machine feature, update the operation-AGV pair feature, and update the AGV-machine pair feature;

[0117] Furthermore, when the objective to be optimized is to minimize the maximum completion time of the job, the expression of the reward in step S3024 is: R t =ct(s t )-ct(s t+1 )

[0118] When the target to be optimized is to maximize the average machine utilization, the expression of the reward in step S3024 is: R t =U ave (t)-U ave (t+1) In the formula, R t is the reward; ct(·) is the completion time at a certain moment; s t is the state of the environment at time t; Uave (·) is the average utilization rate of the machine at a certain moment;

[0119] The beneficial effects of the above scheme are: appropriate reward expressions are selected according to different goals, the agent takes different actions under the environment state according to the target reward value, maximizes the long-term cumulative reward, and evaluates and feeds back the agent's actions to guide the learning process.

[0120] Furthermore, in the process of obtaining the scheduling solution by using the job scheduling model in step S4, the operator-AGV-machine pair with the largest probability distribution value is used as the selection criterion.

[0121] The beneficial effect of the above scheme is: after deep feature extraction, the strategy is continuously optimized and the optimal action is selected in the hope of obtaining the maximum cumulative reward.

[0122] The invention provides an example of flexible job shop scheduling, which covers 3 jobs, 2 AGVs and 3 machines, as shown in Table 1, Table 2 and Table 3. In the table, "-" is used to indicate that a certain operation cannot be processed on a specific machine, or that the subsequent operation of a certain operation is completed on the same machine and does not require AGV transportation. It is assumed that all AGVs can transport all jobs. Table 1

[0123] The above example uses a disjunctive graph to represent the scheduling problem, such as Figure 4 As shown in the figure, the black solid line represents the priority constraint relationship of each operation. The different dotted lines represent the processing machines and AGVs that can be used for each operation.

[0124] Table 2 shows the transfer time of the AGV between the warehouse and each machine. Table 2 layout W <![CDATA[M 1 ]]> <![CDATA[M 2 ]]> <![CDATA[M 3 ]]> W 0 6 5 4 <![CDATA[M 1 ]]> 6 0 2 3 <![CDATA[M 2 ]]> 5 2 0 6 <![CDATA[M 3 ]]> 4 3 6 0

[0125] Table 3 shows a complete scheduling plan, and the corresponding scheduling plan Gantt chart is as follows Figure 7 In addition, Figure 8 The corresponding example solution graph is shown, clearly showing the machine and AGV selected by the agent for the operation. In this example, the final completion time of the job is 17 hours, 41 minutes and 37 seconds. Table 3

[0126] Experiments show that the present invention can utilize its specific architecture and training method to better capture complex patterns and relationships in the data, thereby improving the performance and generalization ability of the model.

[0127] Finally, it should be noted that the parts of the present invention that are not described in detail are all prior art. Those of ordinary skill in the art can understand that the above are only preferred examples of the invention and are not intended to limit the invention. Although the invention is described in detail with reference to the aforementioned examples, those of ordinary skill in the art can still modify the technical solutions recorded in the aforementioned examples, or replace some of the technical features therein with equivalents. Any modifications, equivalent replacements, etc. made within the spirit and principles of the invention should be included in the scope of protection of the invention.

Claims

1. A flexible job shop scheduling method based on deep reinforcement learning, characterized in that: The following steps are involved: S1, taking minimization of the maximum completion time of the job and maximization of the average machine utilization as the optimization objectives, the target model is obtained; S2. Use the disjunctive graph model to represent the flexible job shop scheduling problem and obtain the scheduling status; S3. According to the scheduling state and the target model, an end-to-end learning framework is constructed. The framework combines a multiple attention network for deep feature extraction and a decision network for scalable decision making, and uses a Markov decision process to obtain a job scheduling model; the multiple attention network includes an AGV information attention block, an operation information attention block, and a machine information attention block that are interconnected, and the decision network is trained in a deep reinforcement learning manner; S4. The multi-attention network uses the complex relationship between operations, AGVs, and machines to construct a production-adaptive operation-AGV-machine solution to support the decision-making of the decision network, and uses the job scheduling model to obtain a scheduling schedule to complete flexible job shop scheduling.

2. A flexible job shop scheduling method based on deep reinforcement learning according to claim 1, characterized in that: The expression of the target model in step S1 is: Where FJSPMOD is the target model; Minimize is the minimization function; Makespan is the maximum completion time of the i-th job to complete all operations; max is the maximization function; The time it takes for the i-th job to complete all operations; i is the job number; n is the number of jobs; n i is the total number of operations of the ith job; U ave is the average machine utilization rate; m is the number of machines; k is the machine number; j is the operation number of the i-th job; t ijk is the processing time of the jth operation of the i-th job on the k-th machine; X ijk It is an identifier of whether the jth operation of the ith job can be performed on the kth machine. If yes, it is 1, otherwise, it is 0; time is the current time of the environment.

3. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 1, characterized in that: The expression of the scheduling state in step S2 is: V=(O,M,C,D,G) In the formula, V is the scheduling state, O is the operation node, M is the machine node, C is the conjunction arc, D is the disjunction arc, G is the AGV node, and O is the ij is the jth operation of the ith job; i is the job number; j is the operation number of the ith job; Start is the virtual start node with a processing time of zero; End is the end node.

4. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 1, characterized in that: The step S3 is specifically as follows: S301, obtaining instance data; S302. According to the instance data, scheduling status and target model, the intelligent agent is trained using the Markov decision process, and multiple attention networks are used for operation, AGV and machine feature extraction. The policy distribution and state value are output through the decision network, and the proximal policy optimization algorithm is used to train the intelligent agent. The state, action and reward value are selected and recorded to obtain the job scheduling model.

5. A flexible job shop scheduling method based on deep reinforcement learning according to claim 4, characterized in that: The step S302 is specifically as follows: S3021. Randomly generate training instances according to preset rules, including operation length, operation processing time, and AGV transportation time; S3022, using the scheduling state, input the operation feature into the operation information attention block for calculation, and output the operation node feature; input the AGV feature into the AGV information attention block for calculation, and output the AGV node feature; input the machine feature into the machine information attention block for calculation, and output the machine node feature, respectively average pooling and connecting the operation node feature, AGV node feature and machine node feature to generate the operation global feature, AGV global feature and machine global feature; S3023, inputting the operation global features, AGV global features and machine global features into the strategy value network respectively, and outputting the scheduling strategy and state value; the decision network includes a plurality of parallel strategy value networks, and the strategy value network includes a strategy network for generating strategies and a value network for evaluating the value of the current state; S3024. Use the scheduling status and policy distribution to select the operation-AGV-machine pair, update the operation start time and end time, AGV empty transport time, load transport start time and load transport end time in the operation-AGV-machine pair, use the proximal policy optimization algorithm to train the intelligent agent, calculate the reward according to the target to be optimized, and update the job scheduling model parameters until the model converges.

6. A flexible job shop scheduling method based on deep reinforcement learning according to claim 5, characterized in that: The total loss function in the proximal strategy optimization algorithm is: Where, L clip (θ) is the loss function of the policy network; θ is the parameter of the policy network; min{·} is the minimization function; r t (θ) is the ratio of the new and old strategies; is the advantage function at time t; t is the time; clip(·) is the clipping function; ε is the clipping parameter; π(·) is the new strategy; a t is the action of the agent at time t; θ is the network parameter corresponding to the new strategy; π old (·) is the old strategy; θ old is the network parameter corresponding to the old strategy; L value (φ) is the loss function of the value network; φ is the parameter of the value network; V φ (·) is the state value of the value network; s t is the state of the agent at time t; is the target value at time t, which is the weighted sum of the immediate reward and the future value state; R t is the immediate reward, which indicates the benefit of the current action; γV φ (s t+1 ) represents the possible future return; γ is the discount factor; s t+1 is the new state of the agent; L entropy is the entropy regularization term, which is used to increase the randomness of the strategy and prevent the strategy from converging prematurely; π is the discrete probability distribution of the input, that is, the probability distribution of selecting action a in state s; L PPO (θ, φ) is the total loss function; c1 and c2 are weight hyperparameters; Q(·) is the state-action value function; V(·) is the state value function.

7. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 5, characterized in that: The step S3022 is specifically as follows: Using the scheduling state, the operation features and operation masks are input into the operation information attention block. After linear change and rolling combination operations, the domain information of the source node operation features and the domain information of the target node operation features are generated. The source node operation features and the target node operation features are broadcast and added to generate interaction features. After that, the interaction features are nonlinearly transformed, masked, and softmax normalized. The neighborhood features of the operation nodes are weighted and aggregated using the attention weights to generate new operation features. The operation feature, operation mask, workpiece candidate value, machine feature, machine mask and machine competition index are input into the machine information attention block; through the filter connected between the operation information attention block and the machine information attention block, the machine competition index in the environment is matrix-multiplied with the operation feature matrix to obtain the machine competition variable; then the machine feature and the edge feature between the operation and the machine are linearly transformed, the source node machine feature, the target node machine feature and the edge node feature between the operation and the machine are broadcast and added, and then the attention coefficient between the two machines is nonlinearly transformed, masked and softmax normalized, and the machine nodes with competitive relationship are weightedly aggregated using the attention weight to generate a new machine feature; The operation features, operation masks, workpiece candidate values, AGV features, AGV masks and AGV competition indexes are passed into the AGV information attention block; through the filter connected between the operation information attention block and the AGV information attention block, the AGV competition index in the environment is matrix-multiplied with the operation feature matrix to obtain the AGV competition variable, and then the AGV features and the edge features between the operation and the AGV are linearly transformed, the source node AGV features, the target node AGV features and the edge node features between the operation and the AGV are broadcast and added, and then the attention coefficient between the two AGVs is nonlinearly transformed, masked, and softmax normalized, and the attention weights are used to perform weighted aggregation on the AGV nodes with competing relationships to generate new AGV features.

8. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 7, characterized in that: The S3022 specifically includes: there are multiple operation information attention blocks, AGV information attention blocks and machine information attention blocks, which respectively perform feature calculations on their respective multiple attention heads to generate outputs for each attention head, and then use a splicing mode or an averaging mode to merge the multiple attention head outputs.

9. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 5, characterized in that: The step S3023 is specifically as follows: S30231. Using the policy network, the extracted operation features, AGV features, machine features, operation-AGV pair features, machine-AGV pair features, and global features are input to generate a scalar for each candidate action, and the operation-AGV-machine pair is obtained. The obtained scalar is calculated as follows: In the formula, μ(a t |s t ) is the priority index of action a selected in state s at time t; MLP θ is a multilayer perceptron based on θ; θ is the network parameter of MLP; The output features of the operation information attention block; is the output feature of the AGV information attention block; is the output feature of the machine information attention block; It is the global feature of the FJSP instance; is the output feature of the operator-AGV pair; is the output feature of the operation-machine pair; S30232. Use the softmax function to normalize the priority index scalar and output the probability distribution of the selected action. The calculation formula is: In the formula, π θ (a t |s t ) is the probability of selecting the operation-AGV-machine pair; a t is the action of the agent; s t is the state at time t; exp(·) is an exponential function with a natural constant as the base; μ(a t |s t ) is the priority index of action a selected in state s at time t; b t is the action element in the set of optional actions for the agent; b t is the set of optional actions at time t; θ is the network parameter of MLP.

10. The method for flexible job shop scheduling based on deep reinforcement learning according to claim 5, characterized in that: Step S3024 is specifically as follows: S30241. Update candidate operations according to the operation, AGV and machine index of the selected action. If the current operation is not the last operation of the job, update the candidate operation. Otherwise, mark it as completed. S30242, using the scheduling status, update the AGV transportation time, take the maximum value of the operation and the current idle time of the AGV to update the AGV empty transportation start time, update the AGV load transportation time according to the AGV empty transportation time, take the AGV load transportation end time and the maximum idle time of the machine to update the operation start time, update the operation completion time according to the operation processing time, update the above time information, and obtain the scheduling time schedule; S30243. Update the scheduled maximum completion time according to the operation completion time; S30244. Update the operation status according to the index value of the completed operation, mark the scheduled operation and adjust the remaining operation-AGV-machine relationship matrix, delete the completed operation, update the operation mask, AGV mask, machine mask, operation feature, machine feature, operation-AGV pair feature, AGV-machine pair feature.

Images