Multi-target flexible job shop scheduling method based on graph network and reinforcement learning

Abstract

The invention discloses a multi-target flexible job shop scheduling method based on a graph network and reinforcement learning, which can effectively characterize key features of a production model by using features of a double-graph attention network deep mining process and a machine. The depth characterization capability can not only improve the performance of the model, but also enhance the adaptability of the model to different production models, so that the model can flexibly cope with various unseen production scenes, and shows excellent generalization performance. Besides, in the field of multi-objective optimization, how to balance the relationship between different objectives is an important and challenging problem, so that the model can find an optimal balance point among the multiple objectives by distributing different weights for different optimization objectives, and the method not only can improve the solving efficiency, but also can improve the optimal balance point among the multiple objectives. And moreover, a high-quality Pareto frontier solution set can be generated, and an effective solution is provided for a multi-target flexible job shop scheduling problem.

Description

Technical Field

[0001] The present invention relates to the technical field of flexible job shop scheduling, and in particular to a multi-objective flexible job shop scheduling method based on graph network and reinforcement learning. Background Art

[0002] In the wave of Industry 4.0, the way companies produce, improve and distribute products is undergoing a revolutionary change. This change is driving the manufacturing industry to develop in a faster, smarter and more flexible direction, thus triggering a fundamental leap in corporate production capacity. In order to adapt to the rapid changes in the market and the flexibility requirements of manufacturing systems, many experts and scholars have proposed a series of innovative manufacturing system models, such as multi-agent systems, flexible manufacturing systems and agile manufacturing engineering. The introduction of these new models has undoubtedly accelerated the new process of manufacturing development.

[0003] In this context, the Flexible Job-Shop Scheduling Problem (FJSP) has become a representative challenge in the field of flexible manufacturing. FJSP allows a process to be processed on multiple different machines, which involves not only the sequencing of processes but also the allocation of machines. Compared with the traditional job-shop scheduling problem, FJSP has a more complex topological structure and a larger solution space, which enables it to better adapt to the flexibility and diversity of the relationship between tasks and resources in the new manufacturing paradigm.

[0004] In the current production scheduling field, the main methods used include precise methods and heuristic methods. Precise methods, such as branch and bound and column generation, aim to find the optimal solution, but they often take a lot of computing time when facing multi-objective optimization problems, especially in complex production environments, where this time cost increases significantly. Although heuristic methods give up the guarantee of the optimal solution and can obtain high-quality scheduling solutions in a shorter time, they also face the problems of long computing time and reduced solution quality when dealing with multi-objective problems. Summary of the invention

[0005] The purpose of the present invention is to overcome the shortcomings of the prior art and provide a multi-objective flexible job shop scheduling method based on graph network and reinforcement learning.

[0006] To achieve the above purpose, the technical solution provided by the present invention is:

[0007] A multi-objective flexible job shop scheduling method based on graph network and reinforcement learning, including:

[0008] The process of solving multi-objective flexible job shop scheduling is defined as a Markov decision process;

[0009] The flexible job shop scheduling problem is modeled as a heterogeneous disjunctive graph;

[0010] Use dual-graph attention neural network to adaptively represent process and machine node features to obtain a decision model;

[0011] The decision model is trained by using a proximal strategy optimization method to obtain a trained decision model;

[0012] Multi-objective flexible job shop scheduling is carried out through the trained decision-making model.

[0013] Furthermore, the process of solving multi-objective flexible job shop scheduling is considered as a sequential decision process, where the processes are assigned to the processable machines in each state until all the processes are scheduled;

[0014] The process of solving multi-objective flexible job shop scheduling is defined as a Markov decision process, consisting of the tuple constitute;

[0015] is the state set, and the state of all processes and machines at time t is expressed as

[0016] is an action set, combining process selection and machine allocation into one decision action A decision action is defined as a feasible process machine pair (O ij ,M k ), O ij represents the jth process of the ith job that is allowed to be scheduled, represents the compatible machine k of the jth process of the i-th workpiece, Represents the set of all machines;

[0017] is the state transfer function, indicating that in state S t Next, complete decision action a t Then transition to a new state S t+1 The probability of

[0018] R is the completion of decision action a t The reward is given by state S t Convert to S t+1 The reward is based on the weighted combination, r t ∈R is the production cycle reward r ms With load balancing reward r std Based on the weight combination, i.e. r t =(1-β)r ms +βr std, where β ranges from [0,1]; Indicates in s t Calculate O ij The estimated completion time of process O ij When not scheduled, Indicates O ij On all compatible machines The minimum processing time in . If the scheduling has been completed, the actual processing time of the corresponding machine is used; and All machines at the current time t and at time t-1 respectively standard deviation of processing time;

[0019] γ is a discount factor with a value range of [0,1]. Values ​​close to 1 focus more on long-term cumulative rewards, while values ​​close to 0 focus more on short-term rewards.

[0020] Furthermore, the structure of the heterogeneous disjunctive graph H is Where O is the process node set; M is the machine node set, and C is the directed arc set. Scheduling state for the flexible job shop scheduling problem;

[0021] Each state s t Represented as a heterogeneous graph in Changes dynamically during the solution process.

[0022] Furthermore, a dual graph attention neural network is used to adaptively represent process and machine node features, including aggregating process features based on a multi-order graph attention neural network and aggregating machine features based on a graph attention neural network.

[0023] Furthermore, the aggregation process features based on the multi-order graph attention neural network include:

[0024] By adding an attention mechanism to adaptively aggregate process node features in multiple order fields, the feature information of each hop neighbor is weighted:

[0025]

[0026] In the formula, m ij,l represents the weight of the l-th order feature of the ij node, represents the l-th order feature of the ij node, and L represents the total number of levels;

[0027]

[0028] In the formula, h ij represents the ij node feature, and σ represents the nonlinear activation function.

[0029] Furthermore, the machine features are aggregated based on the graph attention neural network, including:

[0030] In the machine feature embedding of the heterogeneous disjunctive graph H, process O ij On compatible machines M k The processing time p ijk By taking it as the characteristic E of the process machine arc ijk To express;

[0031] The attention coefficient of a machine node is calculated as:

[0032]

[0033] In the formula, v k For machine M k The initial characteristics, W M is the linear conversion parameter of the machine, LeakyReLU is the activation function, is the transpose of the trainable weight parameters;

[0034] Then, e kk For machine M k Its own attention coefficient, all e ijk With e kk Normalize to obtain the normalized attention coefficient α ijk and α kk ;

[0035] Finally, the embedding v' is calculated by aggregating the features of adjacent operations and itself k Machine, computing machine M k Features of v' k The aggregation function is:

[0036]

[0037] In the formula, Indicates machine M k And all the processes that can be handled.

[0038] Furthermore, the decision model is trained by a proximal strategy optimization method, including:

[0039] For each time step t, a t =(O ij ,M k )∈A t For each actionable pair, the corresponding process, machine and average pooling feature embedding are concatenated and input into the MLP to obtain its state s t Select action a t The probability scalar parameter P(a t ,s t),as follows:

[0040]

[0041] In the formula, MLP means using MLP neural network to construct decision network, ω is the network parameter, and h ij and v' k The features after passing through L layers of neural network, h G for and Global features after average pooling;

[0042] For all P(a t ,s t ) Normalize and calculate each a t The probability of outputting the corresponding probability distribution is:

[0043]

[0044] The algorithm is based on the Actor-Critic framework, calculates the policy gradient, evaluates and updates the current policy; its optimization objective function is:

[0045]

[0046] In the formula, π θ Indicates that the new policy network is in state s t Next select action a t The probability of old Indicates the old policy network status s t Next select action a t The probability of A t A(s,a) represents the advantage function of state s action a, clip represents the truncation operation, ∈ is a truncation range;

[0047] The Critic network is used as the evaluation module, using the global feature h G As the input of the network,

[0048]

[0049] In the formula, Indicates the current state S t evaluation, θ represents the parameters of the evaluation network.

[0050] Compared with the prior art, the principles and advantages of this technical solution are as follows:

[0051] 1. Using the dual-graph attention network to deeply mine the features of the process and the machine can effectively characterize the key features of the production model. This deep characterization capability not only improves the performance of the model, but also enhances the model's adaptability to different production models, enabling it to flexibly respond to various unseen production scenarios and demonstrate excellent generalization performance.

[0052] 2. In the field of multi-objective optimization, how to balance the relationship between different objectives is an important and challenging problem. This technical solution enables the model to find an optimal balance point between multiple objectives by assigning different weights to different optimization objectives. This method can not only improve the solution efficiency, but also generate high-quality Pareto frontier solution sets, providing an effective solution to the multi-objective flexible job shop scheduling problem. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the services required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0054] Figure 1 It is a flow chart of the multi-objective flexible job shop scheduling method based on graph network and reinforcement learning of the present invention;

[0055] Figure 2 It is a principle diagram of the multi-objective flexible job shop scheduling method based on graph network and reinforcement learning of the present invention;

[0056] Figure 3 Heterogeneous disjunctive graph for the flexible job shop scheduling problem. DETAILED DESCRIPTION

[0057] The present invention will be further described below in conjunction with specific embodiments:

[0058] like Figure 1 and Figure 2 As shown, the multi-objective flexible job shop scheduling method based on graph network and reinforcement learning described in this embodiment includes the following steps:

[0059] S1. The process of solving multi-objective flexible job shop scheduling is regarded as a sequential decision process. In each state, the process is assigned to the processable machine until all processes are scheduled. Therefore, the process of solving multi-objective flexible job shop scheduling is defined as a Markov decision process, which consists of the tuple constitute;

[0060] in,

[0061] is the state set, and the state of all processes and machines at time t is expressed as

[0062] is an action set, combining process selection and machine allocation into one decision action A decision action is defined as a feasible process machine pair (O ij ,M k ), O ij represents the jth process of the ith job that is allowed to be scheduled, represents the compatible machine k of the jth process of the i-th workpiece, Represents the set of all machines;

[0063] is the state transfer function, indicating that in state S t Next, complete decision action a t Then transition to a new state S t+1 The probability of

[0064] R is the completion of decision action a t The reward is given by state S t Convert to S t+1 The reward is based on the weighted combination, r t ∈R is the production cycle reward r ms With load balancing reward r std Based on the weight combination, i.e. r t =(1-β)r ms +βr std , where β ranges from [0,1]; Indicates in s t Calculate O ij The estimated completion time of process O ij When not scheduled, Indicates O ij On all compatible machines If the scheduling has been completed, the actual processing time of the corresponding machine is used; r std = and All machines at the current time t and at time t-1 respectively standard deviation of processing time;

[0065] γ is a discount factor with a value range of [0,1]. Values ​​close to 1 focus more on long-term cumulative rewards, while values ​​close to 0 focus more on short-term rewards.

[0066] S2, modeling the flexible job shop scheduling problem as a heterogeneous disjunctive graph;

[0067] like Figure 3 As shown, the structure of the heterogeneous disjunctive graph H is Where O is the process node set; M is the machine node set, and C is the directed arc set. Scheduling state for the flexible job shop scheduling problem;

[0068] Each state s t Represented as a heterogeneous graph in Changes dynamically during the solution process.

[0069] S3, using dual-graph attention neural network to adaptively represent process and machine node features to obtain a decision model;

[0070] This step includes aggregating process features based on a multi-order graph attention neural network and aggregating machine features based on a graph attention neural network;

[0071] Among them, the aggregation process features based on the multi-order graph attention neural network include:

[0072] By adding an attention mechanism to adaptively aggregate process node features in multiple order fields, the feature information of each hop neighbor is weighted:

[0073]

[0074] In the formula, m ij,l represents the weight of the k-th order feature of the ij node, represents the l-th order feature of the ij node, and L represents the total number of levels;

[0075]

[0076] In the formula, h ij represents the ij node feature, and σ represents the nonlinear activation function.

[0077] Aggregate machine features based on graph attention neural network, including:

[0078] In the machine feature embedding of the heterogeneous disjunctive graph H, process O ij On compatible machines M k The processing time p ijk By taking it as the characteristic E of the process machine arc ijk To express;

[0079] The attention coefficient of a machine node is calculated as:

[0080]

[0081] In the formula, v k For machine M k The initial characteristics, WM is the linear conversion parameter of the machine, LeakyReLU is the activation function, is the transpose of the trainable weight parameters;

[0082] Then, e kk For machine M k Its own attention coefficient, all e ijk With e kk Normalize to obtain the normalized attention coefficient α ijk and α kk ;

[0083] Finally, the embedding v' is calculated by aggregating the features of adjacent operations and itself k Machine, computing machine M k Features of v' k The aggregation function is:

[0084]

[0085] In the formula, Indicates machine M k And all the processes that can be handled.

[0086] S4. Train the decision model through the proximal strategy optimization method to obtain a trained decision model;

[0087] The process of training a decision model through proximal policy optimization methods includes:

[0088] For each time step t, a t =(O ij ,M k )∈A t For each actionable pair, the corresponding process, machine and average pooling feature embedding are concatenated and input into the MLP to obtain its state s t Select action a t The probability scalar parameter P(a t ,s t ),as follows:

[0089]

[0090] In the formula, MLP means using MLP neural network to construct decision network, ω is the network parameter, and h ij and v' k The features after passing through L layers of neural network, h G for and Global features after average pooling;

[0091] For all P(a t ,s t ) Normalize and calculate each a t The probability of outputting the corresponding probability distribution is:

[0092]

[0093] The algorithm is based on the Actor-Critic framework, calculates the policy gradient, evaluates and updates the current policy; its optimization objective function is:

[0094]

[0095] In the formula, π θ Indicates that the new policy network is in state s t Next select action a t The probability of old Indicates the old policy network status s t Next select action a t The probability of A t A(s,a) represents the advantage function of state s action a, clip represents the truncation operation, ∈ is a truncation range;

[0096] The Critic network is used as the evaluation module, using the global feature h G As the input of the network,

[0097]

[0098] In the formula, Indicates the current state S t evaluation, θ represents the parameters of the evaluation network.

[0099] S5. Finally, the trained decision model is used to perform multi-objective flexible job shop scheduling.

[0100] The embodiments described above are only preferred embodiments of the present invention and are not intended to limit the scope of implementation of the present invention. Therefore, all changes made according to the shape and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A multi-objective flexible job shop scheduling method based on graph network and reinforcement learning, characterized in that: include: The process of solving multi-objective flexible job shop scheduling is defined as a Markov decision process; The flexible job shop scheduling problem is modeled as a heterogeneous disjunctive graph; Use dual-graph attention neural network to adaptively represent process and machine node features to obtain a decision model; The decision model is trained by using a proximal strategy optimization method to obtain a trained decision model; Multi-objective flexible job shop scheduling is carried out through the trained decision-making model.

2. The multi-objective flexible job shop scheduling method based on graph network and reinforcement learning according to claim 1 is characterized in that: The multi-objective flexible job shop scheduling process is considered as a sequential decision-making process, where the processes are assigned to the machines that can be processed in each state until all the processes are scheduled; The process of solving multi-objective flexible job shop scheduling is defined as a Markov decision process, consisting of the tuple constitute; is the state set, and the state of all processes and machines at time t is expressed as is an action set, combining process selection and machine allocation into one decision action A decision action is defined as a feasible process machine pair (O ij ,M k ), O ij represents the jth process of the ith job that is allowed to be scheduled, represents the compatible machine k of the jth process of the i-th workpiece, Represents the set of all machines; is the state transfer function, indicating that in state S t Next, complete decision action a t Then transition to a new state S t+1 probability; R is the completion of decision action a t The reward is given by state S t Convert to S t+1 The reward is based on the weighted combination, r t ∈R is the reward r for the production cycle ms With load balancing reward r std Based on the weight combination, i.e. r t =(1-β)r ms +βr std , where β ranges from [0,1]; Cbl(O ij ,s t ) means in s t Calculate O ij The estimated completion time of process O ij When not scheduled, Indicates O ij On all compatible machines The minimum processing time in . If the scheduling has been completed, the actual processing time of the corresponding machine is used; and All machines at the current time t and at time t-1 respectively standard deviation of processing time; γ is a discount factor with a value range of [0,1]. Values ​​close to 1 focus more on long-term cumulative rewards, while values ​​close to 0 focus more on short-term rewards.

3. The multi-objective flexible job shop scheduling method based on graph network and reinforcement learning according to claim 2 is characterized in that: The structure of the heterogeneous disjunctive graph H is Where O is the process node set; M is the machine node set, and C is the directed arc set. Scheduling state for the flexible job shop scheduling problem; Each state s t Represented as a heterogeneous graph in Changes dynamically during the solution process.

4. The multi-objective flexible job shop scheduling method based on graph network and reinforcement learning according to claim 3 is characterized in that: A dual graph attention neural network is used to adaptively represent process and machine node features, including aggregating process features based on a multi-order graph attention neural network and aggregating machine features based on a graph attention neural network.

5. The multi-objective flexible job shop scheduling method based on graph network and reinforcement learning according to claim 4 is characterized in that: The characteristics of the aggregation process based on the multi-order graph attention neural network include: By adding an attention mechanism to adaptively aggregate process node features in multiple order fields, the feature information of each hop neighbor is weighted: In the formula, m ij,l represents the weight of the l-th order feature of the ij node, represents the l-th order feature of the ij node, and L represents the total number of levels; In the formula, h ij represents the ij node feature, and σ represents the nonlinear activation function.

6. The multi-objective flexible job shop scheduling method based on graph network and reinforcement learning according to claim 5 is characterized in that: Aggregate machine features based on graph attention neural network, including: In the machine feature embedding of the heterogeneous disjunctive graph H, process O ij On compatible machines M k The processing time p ijk By taking it as the characteristic E of the process machine arc ijk To express; The attention coefficient of the machine node is calculated as: In the formula, v k For machine M k The initial characteristics, W M is the linear conversion parameter of the machine, LeakyReLU is the activation function, is the transpose of the trainable weight parameters; Then, e kk For machine M k Its own attention coefficient, all e ijk With e kk Normalize to obtain the normalized attention coefficient α ijk and α kk ; Finally, the embedding v' is calculated by aggregating the features of adjacent operations and itself k Machine, computing machine M k Features of v' k The aggregation function is: In the formula, Indicates machine M k And all the processes that can be handled.

7. The multi-objective flexible job shop scheduling method based on graph network and reinforcement learning according to claim 6 is characterized in that: The decision model is trained through proximal policy optimization methods, including: For each time step t, a t =(O ij ,M k )∈A t For each actionable pair, the corresponding process, machine and average pooling feature embedding are concatenated and input into the MLP to obtain its state s t Select action a t The probability scalar parameter P(a t ,s t ),as follows: In the formula, MLP means using MLP neural network to construct decision network, ω is the network parameter, and h ij and v' k The features after passing through L layers of neural network, h G for and Global features after average pooling; For all P(a t ,s t ) Normalize and calculate each a t The probability of outputting the corresponding probability distribution is: The algorithm is based on the Actor-Critic framework, calculates the policy gradient, evaluates and updates the current policy; its optimization objective function is: In the formula, π θ Indicates that the new policy network is in state s t Next select action a t The probability of old Indicates the old policy network status s t Next select action a t The probability of A t A(s,a) represents the advantage function of state s action a, clip represents the truncation operation, ∈ is a truncation range; The Critic network is used as the evaluation module, using the global feature h G As the input of the network, In the formula, Indicates the current state S t evaluation, θ represents the parameters of the evaluation network.

Images