Flexible job shop scheduling method based on deep reinforcement learning algorithm

Abstract

The invention discloses a flexible job shop scheduling method based on a deep reinforcement learning algorithm, and particularly relates to the field of flexible job shop scheduling, and the method comprises the steps: S1, constructing a job set and a job set production and processing total time model; s2, setting corresponding constraint conditions, and constructing a flexible job shop scheduling model; s3, according to the job set information and the workshop resource related information, representing the job set and the machine information by using a disjunction graph and a hypergraph respectively as outputs, and substituting the outputs into a graph isomorphic network (GIN) and a hypergraph convolutional network (HGCN) to complete information extraction; s4, implementing a decision by adopting a near-end optimization strategy; solving the flexible job shop scheduling model to obtain a production processing scheme; s5, training a network model of a deep reinforcement learning method to obtain an optimal action network model, and performing production processing on the operation set according to the obtained production processing scheme; and an optimal production and processing scheme with shortest time consumption is obtained, and the production and processing efficiency of the operation set is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of flexible job shop scheduling, and more specifically, to a flexible job shop scheduling method based on deep reinforcement learning algorithm; Background Art

[0002] With the development of artificial intelligence, digital automatic production has become a trend in factory upgrading and transformation. Flexible job shop scheduling, as a key decision-making activity in workshop management, plays a key role in workshop-level production management.

[0003] Traditional single-workshop scheduling methods have certain limitations in dealing with distributed workshop scheduling problems. Existing workshop scheduling methods usually require a lot of computing time to solve a satisfactory approximate optimal solution. Neural networks without reinforcement learning are difficult to cope with ever-changing demands and changing processing environments. Summary of the invention

[0004] To achieve the above object, the present invention provides the following technical solution: a flexible job shop scheduling method based on deep reinforcement learning algorithm, the steps of which include:

[0005] S1: Construct the model of the operation set and the total production and processing time of the operation set;

[0006] S2: Set corresponding constraints and build a flexible job shop scheduling model;

[0007] S3: According to the job set information and workshop resource related information, the job set and machine information are represented by disjunctive graph and hypergraph respectively as output and brought into the graph isomorphism network (GIN) and hypergraph convolutional network (HGCN) to complete the information extraction;

[0008] S4: adopting the proximal optimization strategy to achieve decision-making; solving the flexible job shop scheduling model to obtain a production and processing plan;

[0009] S5: Train the network model of the deep reinforcement learning method, obtain the optimal action network model, and produce and process the job set according to the obtained production and processing plan; obtain the optimal production and processing plan with the shortest time consumption, and improve the production and processing efficiency of the job set;

[0010] In a preferred embodiment, the specific steps of constructing the operation set and the total production and processing time model of the operation set in S1 are: constructing the operation set J = {J1, J2, ..., J i ,…,J n}, where J i represents the Jth job, and O ij represents the i-th job J iThe jth process, n i represents the i-th job J i The total number of processes, n represents the total number of jobs in the job set J;

[0011] In a preferred embodiment, the process of constructing a flexible job shop scheduling model is as follows:

[0012] S2.1. Use formula (1) to construct the objective function F of the flexible job shop scheduling model:

[0013] F=minC max (1)

[0014] In formula (1), C max represents the total production and processing time of job set J;

[0015] S2.2, use equations (2) to (5) to construct the constraints of the single-objective flexible job shop scheduling model;

[0016]

[0017] In formula (2) to formula (5), M = {1, 2, 3...}, where Mk represents the k-th machine, Mij is the set of machines available for the processing step Oij, and Xijk represents the i-th operation J i The jth process O ij Whether it is processed on machine k, if yes, set Xijk = 1, if not, set Xijk = 0; C ij represents the completion time of process Oij, tijk represents the processing time of process Oij on machine Mk, A i represents the arrival time of the job, Yijlg represents whether process Oij is the lth job J l The previous process of the g-th process Olg, if yes, let Yijlg = 1, otherwise let Yijlg = -1;

[0018] In a preferred embodiment, the steps of extracting information in S3 are as follows:

[0019] S3.1, disjunctive graph G = (V, C ∪ D) is used to represent, where V represents all operations, that is, all processes O of J ij , C is the reflection of each J i Process O ij The edge set of constraints between them indicates the sequence of the processes of each job; D reflects the edge set of the sequence of process operations on machine Mk, reflecting the sequence of machine processing processes; and the adjacency matrix A is obtained from G G , the adjacency matrix A G is an n*n matrix with an initial value of 0. ijAssign a unique number (from 0 to n-1), fill in the edge sets C and D, and mark them as 1 in the adjacency matrix if there is an edge connection;

[0020] S3.2. The estimated processing time after selecting the workpiece and the workpiece completion mark are used as the initial features of each node The estimated processing time is the sum of the previous job scheduling time and the current selected job scheduling time; the workpiece completion flag is 1, otherwise it is 0;

[0021] S3.3, use the aggregation function to sum up the features of neighboring nodes; splice the aggregated features with the features of the current node; calculate the set N(v) of neighboring nodes for each node;

[0022]

[0023] in is the feature representation of node u at layer l; N(v) is the set of neighbor nodes of node v; MLP (l+1) is the l+1th layer of the multilayer perceptron; ∈ (l) is a learnable scalar parameter used to enhance the representation capability of the model; It means to sum and pool the embeddings of all nodes to get the global representation of the entire graph;

[0024] S3.4. Apply the above operation to all nodes in the graph and update their feature representations. Repeat the above process for several layers, and share the parameters of each layer.

[0025] S3.5, after multiple layers of processing, each node has an extracted feature vector;

[0026] S3.6, hypergraph HG = (v, e), where v is a vertex set, and vertices represent job operations. e is a hyperedge set, and hyperedges represent the entire process of completing a job; D v Represents the degree matrix of the vertex; the number of hypergraph convolution layers is 4 and the dimensions are 256, 256, 256, and 128;

[0027] And get the adjacency matrix A from HG HG , the adjacency matrix A HG is an n*n matrix with an initial value of 0. ij Assign a unique number (from 0 to n-1), fill in the edge set e, and mark it as 1 in the adjacency matrix if there is an edge connection;

[0028] S3.7. The estimated processing time after selecting the workpiece and the workpiece completion mark will be used as the initial features of each node As the initial feature vector of the hypergraph convolutional network; the vertex feature matrix X∈R |v|×2, where |v| is the number of vertices and 2 represents the feature dimension of each vertex; for the adjacency matrix A HG Perform normalization processing; the normalization formula is: Where D v ;

[0029] S3.8, Convolution operation: The adjacency matrix is ​​applied to the feature matrix of the vertex to aggregate the features of neighboring vertices; for hypergraph convolution, the update rule of convolution is: Among them, H ′ is the updated feature matrix after convolution; σ is the ReLU function, and W is the learnable weight matrix;

[0030] S3.9, apply the above operation to all nodes in the graph and update their feature representations; repeat the above process for several layers, and share the parameters of each layer;

[0031] S3.10, after multiple layers of processing, each node has an extracted feature vector;

[0032] In a preferred embodiment, the step of training the network model of the deep reinforcement learning method in S5 is as follows:

[0033] The new feature matrix is ​​used as input, and the proximal optimization strategy is used to achieve decision-making; the Sctor network, the Critic network and the experience replay pool are connected in sequence to construct a neural network model for deep reinforcement learning; wherein the Sctor network and the Critic network are composed of four fully connected layers with sizes of 256, 256, 256 and 64 respectively;

[0034] S5.1. Input the feature vector into the Sctor network to obtain the action probability π(a t |s t );

[0035]

[0037] MLP is a multi-layer perceptron that maps input to a scalar value. The state is s t The features extracted from the kth layer of GIN under ′(K) (s t ) is the state s t The features extracted from the kth layer of HGCN below are used to select actions from the action space using a greedy strategy;

[0038] S5.2. Execute the selected action a t , get the next moment state information S t+1 , and obtain immediate reward r through environmental feedbackt And the state of the next moment S_t+1, (S t ,a t ,r t ,S t+1 ) is stored as experience in the experience replay pool;

[0039] S5.2.1, the reward is estimated by the difference in expected completion time after executing action a_t, that is,

[0040] S5.3. Extract 128 experiences from the experience replay pool to update the neural network model of deep reinforcement learning;

[0041] S5.3.1. Calculate TD target:

[0042]

[0043] Among them, γ0 is the initial discount factor, t is the current time step, T is the total number of time steps, r t+k is the reward at the kth time step, s t+n is the state at time step t+n, π target is the target strategy, Q target is the output of the target Critic network, γ t+k is the dynamic discount factor at time step t+k;

[0044] S5.3.2. Update the Sctor network parameters. The loss of the Sctor network is:

[0045]

[0046] S5.3.3. Update the Critic network parameters. The loss of the Critic network is:

[0047]

[0048] S5.4, repeat S5.1-S5.3 until all workpieces are scheduled;

[0049] S5.5. Extract 128 experiences from the experience replay pool to update the neural network model of deep reinforcement learning;

[0050] S5.6. Repeat S5.1-S5.5 with the updated deep reinforcement learning neural network until the maximum number of training rounds is reached, thereby completing the training of the deep reinforcement learning neural network model.

[0051] Technical effects and advantages of the present invention:

[0052] 1. The present invention constructs a flexible job shop scheduling model, substitutes a graph isomorphism network (GIN) and a hypergraph convolutional network (HGCN) to complete information extraction, trains the network model of the deep reinforcement learning method, and obtains the optimal production and processing plan with the shortest time, thereby improving the production and processing efficiency of the job set. DETAILED DESCRIPTION

[0053] The present invention provides a flexible job shop scheduling method based on deep reinforcement learning algorithm, the steps of which are

[0054] Steps include:

[0055] S1: Construct the model of the operation set and the total production and processing time of the operation set;

[0056] S2: Set corresponding constraints and build a flexible job shop scheduling model;

[0057] S3: According to the job set information and workshop resource related information, the job set and machine information are represented by disjunctive graph and hypergraph respectively as output and brought into the graph isomorphism network (GIN) and hypergraph convolutional network (HGCN) to complete the information extraction;

[0058] S4: adopting the proximal optimization strategy to achieve decision-making; solving the flexible job shop scheduling model to obtain a production and processing plan;

[0059] S5: Train the network model of the deep reinforcement learning method, obtain the optimal action network model, and produce and process the job set according to the obtained production and processing plan; obtain the optimal production and processing plan with the shortest time consumption, and improve the production and processing efficiency of the job set;

[0060] In a preferred embodiment, the specific steps of constructing the operation set and the total production and processing time model of the operation set in S1 are: constructing the operation set J = {J1, J2, ..., J i ,…,J n}, where J i represents the Jth job, and O ij represents the i-th job J i The jth process, n i represents the i-th job J i The total number of processes, n represents the total number of jobs in the job set J;

[0061] In a preferred embodiment, the process of constructing a flexible job shop scheduling model is as follows:

[0062] S2.1. Use formula (1) to construct the objective function F of the flexible job shop scheduling model:

[0063] F=minCmax (1)

[0064] In formula (1), C max represents the total production and processing time of job set J;

[0065] S2.2, use equations (2) to (5) to construct the constraints of the single-objective flexible job shop scheduling model;

[0066]

[0067] In formula (2) to formula (5), M = {1, 2, 3...}, where Mk represents the k-th machine, Mij is the set of machines available for the processing step Oij, and Xijk represents the i-th operation J i The jth process O ij Whether it is processed on machine k, if yes, set Xijk = 1, if not, set Xijk = 0; C ij represents the completion time of process Oij, tijk represents the processing time of process Oij on machine Mk, A i represents the arrival time of the job, Yijlg represents whether process Oij is the lth job J l The previous process of the g-th process Olg, if yes, let Yijlg = 1, otherwise let Yijlg = -1;

[0068] In a preferred embodiment, the steps of extracting information in S3 are as follows:

[0069] S3.1, disjunctive graph G = (V, C ∪ D) is used to represent, where V represents all operations, that is, all processes O of J ij , C is the reflection of each J i Process O ij The edge set of constraints between them represents the sequence of the processes of each job; D reflects the edge set of the sequence of process operations on machine Mk, reflecting the sequence of machine processing processes; and the adjacency matrix A is obtained from G G , the adjacency matrix A G is an n*n matrix with an initial value of 0. ij Assign a unique number (from 0 to n-1), fill in the edge sets C and D, and mark them as 1 in the adjacency matrix if there is an edge connection;

[0070] S3.2. The estimated processing time after selecting the workpiece and the workpiece completion mark are used as the initial features of each node The estimated processing time is the sum of the previous job scheduling time and the current selected job scheduling time; the workpiece completion flag is 1, otherwise it is 0;

[0071] S3.3, use the aggregation function to sum up the features of neighboring nodes; splice the aggregated features with the features of the current node; calculate the set N(v) of neighboring nodes for each node;

[0072]

[0073] in is the feature representation of node u at layer l; N(v) is the set of neighbor nodes of node v; MLP (l+1) is the l+1th layer of the multilayer perceptron; ∈ (l) is a learnable scalar parameter used to enhance the representation capability of the model; It means to sum and pool the embeddings of all nodes to get the global representation of the entire graph;

[0074] S3.4. Apply the above operation to all nodes in the graph and update their feature representations. Repeat the above process for several layers, and share the parameters of each layer.

[0075] S3.5, after multiple layers of processing, each node has an extracted feature vector;

[0076] S3.6, hypergraph HG = (v, e), where v is a vertex set, and vertices represent job operations. e is a hyperedge set, and hyperedges represent the entire process of completing a job; D v Represents the degree matrix of the vertex; the number of hypergraph convolution layers is 4 and the dimensions are 256, 256, 256, and 128;

[0077] And get the adjacency matrix A from HG HG , the adjacency matrix A HG is an n*n matrix with an initial value of 0. ij Assign a unique number (from 0 to n-1), fill in the edge set e, and mark it as 1 in the adjacency matrix if there is an edge connection;

[0078] S3.7. The estimated processing time after selecting the workpiece and the workpiece completion mark will be used as the initial features of each node As the initial feature vector of the hypergraph convolutional network; the vertex feature matrix X∈R |v|×2 , where |v| is the number of vertices and 2 represents the feature dimension of each vertex; for the adjacency matrix A HG Perform normalization processing; the normalization formula is: Where D v ;

[0079] S3.8, Convolution operation: The adjacency matrix is ​​applied to the feature matrix of the vertex to aggregate the features of neighboring vertices; for hypergraph convolution, the update rule of convolution is: Among them, H ′is the updated feature matrix after convolution; σ is the ReLU function, and W is the learnable weight matrix;

[0080] S3.9, apply the above operation to all nodes in the graph and update their feature representations; repeat the above process for several layers, and share the parameters of each layer;

[0081] S3.10, after multiple layers of processing, each node has an extracted feature vector;

[0082] In a preferred embodiment, the step of training the network model of the deep reinforcement learning method in S5 is as follows:

[0083] The new feature matrix is ​​used as input, and the proximal optimization strategy is used to achieve decision-making; the Sctor network, the Critic network and the experience replay pool are connected in sequence to construct a neural network model for deep reinforcement learning; wherein the Sctor network and the Critic network are composed of four fully connected layers with sizes of 256, 256, 256 and 64 respectively;

[0084] S5.1. Input the feature vector into the Sctor network to obtain the action probability π(a t |s t );

[0085]

[0087] MLP is a multi-layer perceptron that maps input to a scalar value. The state is s t The features extracted from the kth layer of GIN under ′(K) (s t ) is the state s t The features extracted from the kth layer of HGCN below are used to select actions from the action space using a greedy strategy;

[0088] S5.2. Execute the selected action a t , get the next moment state information S t+1 , and obtain immediate reward r through environmental feedback t And the state of the next moment S_t+1, (S t ,a t ,r t ,S t+1 ) is stored as experience in the experience replay pool;

[0089] S5.2.1, the reward is estimated by the difference in expected completion time after executing action a_t, that is,

[0090] S5.3. Extract 128 experiences from the experience replay pool to update the neural network model of deep reinforcement learning;

[0091] S5.3.1. Calculate TD target:

[0092]

[0093] Among them, γ0 is the initial discount factor, t is the current time step, T is the total number of time steps, r t+k is the reward at the kth time step, s t+n is the state at time step t+n, π target is the target strategy, Q target is the output of the target Critic network, γ t+k is the dynamic discount factor at time step t+k;

[0094] S5.3.2. Update the Sctor network parameters. The loss of the Sctor network is:

[0095]

[0096] S5.3.3. Update the Critic network parameters. The loss of the Critic network is:

[0097]

[0098] S5.4, repeat S5.1-S5.3 until all workpieces are scheduled;

[0099] S5.5. Extract 128 experiences from the experience replay pool to update the neural network model of deep reinforcement learning;

[0100] S5.6. Repeat S5.1-S5.5 with the updated deep reinforcement learning neural network until the maximum number of training rounds is reached, thereby completing the training of the deep reinforcement learning neural network model.

[0101] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A flexible job shop scheduling method based on deep reinforcement learning algorithm, the steps comprising: S1: Construct the model of the operation set and the total production and processing time of the operation set; S2: Set corresponding constraints and build a flexible job shop scheduling model; S3: According to the job set information and workshop resource related information, the job set and machine information are represented by disjunctive graph and hypergraph respectively as output and brought into the graph isomorphism network (GIN) and hypergraph convolutional network (HGCN) to complete the information extraction; S4: adopting the proximal optimization strategy to achieve decision-making; solving the flexible job shop scheduling model to obtain a production and processing plan; S5: Train the network model of the deep reinforcement learning method to obtain the optimal action network model to produce and process the job set according to the obtained production and processing plan; obtain the optimal production and processing plan with the shortest time consumption to improve the production and processing efficiency of the job set.

2. The flexible job shop scheduling method based on deep reinforcement learning algorithm according to claim 1, characterized in that: The specific steps of constructing the operation set and the total production and processing time model of the operation set in S1 are as follows: constructing the operation set J = {J1, J2, ..., J i ,…,J n }, where J i represents the Jth job, and J i = {O i1 ,O i2 ,…,O ij ,…,O ini },O ij represents the i-th job J i The jth process, n i represents the i-th job J i The total number of processes, n represents the total number of jobs in job set J.

3. The flexible job shop scheduling method based on deep reinforcement learning algorithm according to claim 1, characterized in that; The process of building a flexible job shop scheduling model is as follows: S2.

1. Use formula (1) to construct the objective function F of the flexible job shop scheduling model: F=minC max (1) In formula (1), C max represents the total production and processing time of job set J; S2.2, use equations (2) to (5) to construct the constraints of the single-objective flexible job shop scheduling model; In formula (2) to formula (5), M = {1, 2, 3...}, where Mk represents the k-th machine, Mij is the set of machines available for the processing step Oij, and Xijk represents the i-th operation J i The jth process O ij Whether it is processed on machine k, if yes, set Xijk = 1, if not, set Xijk = 0; C ij represents the completion time of process Oij, tijk represents the processing time of process Oij on machine Mk, A i represents the arrival time of the job, Yijlg represents whether process Oij is the lth job J l The previous process of the g-th process Olg, if it is, let Yijlg = 1, otherwise let Yijlg = -1.

4. The flexible job shop scheduling method based on deep reinforcement learning algorithm according to claim 1, characterized in that; The steps for extracting information in S3 are as follows: S3.1, disjunctive graph G = (V, C ∪ D) is used to represent, where V represents all operations, that is, all processes O of J ij , C is the reflection of each J i Process O ij The edge set of constraints between them represents the sequence of the processes of each job; D reflects the edge set of the sequence of process operations on machine Mk, reflecting the sequence of machine processing processes; and the adjacency matrix A is obtained from G G , the adjacency matrix A G is an n*n matrix with an initial value of 0. ij Assign a unique number (from 0 to n-1), fill in the edge sets C and D, and mark them as 1 in the adjacency matrix if there is an edge connection; S3.

2. The estimated processing time after selecting the workpiece and the workpiece completion mark are used as the initial features of each node The estimated processing time is the sum of the previous job scheduling time and the current selected job scheduling time; the workpiece completion flag is 1, otherwise it is 0; S3.3, use the aggregation function to sum up the features of neighbor nodes; Concatenate the aggregated features with the features of the current node; calculate the set of neighboring nodes N(v) for each node; in is the feature representation of node u at layer l; N(v) is the set of neighbor nodes of node v; MLP (l+1) is the l+1th layer of the multilayer perceptron; ∈ (l) is a learnable scalar parameter used to enhance the representation capability of the model; It means to sum and pool the embeddings of all nodes to get the global representation of the entire graph; S3.

4. Apply the above operation to all nodes in the graph and update their feature representations. Repeat the above process for several layers, and share the parameters of each layer. S3.5, after multiple layers of processing, each node has an extracted feature vector; S3.6, hypergraph HG = (v, e), where v is a vertex set, and vertices represent job operations. e is a hyperedge set, and hyperedges represent the entire process of completing a job; D v Represents the degree matrix of the vertex; the number of hypergraph convolution layers is 4 and the dimensions are 256, 256, 256, and 128; And get the adjacency matrix A from HG HG , the adjacency matrix A HG is an n*n matrix with an initial value of 0. ij Assign a unique number (from 0 to n-1), fill in the edge set e, and mark it as 1 in the adjacency matrix if there is an edge connection; S3.

7. The estimated processing time after selecting the workpiece and the workpiece completion mark will be used as the initial features of each node As the initial feature vector of the hypergraph convolutional network; vertex Feature matrix X∈R |v|×2 , where |v| is the number of vertices and 2 represents the feature dimension of each vertex; for the adjacency matrix A HG Perform normalization processing; the normalization formula is: Where D v ; S3.8, Convolution operation: The adjacency matrix is ​​applied to the feature matrix of the vertex to aggregate the features of neighboring vertices; for hypergraph convolution, the update rule of convolution is: Among them, H ′ is the updated feature matrix after convolution; σ is the ReLU function, and W is the learnable weight matrix; S3.9, apply the above operation to all nodes in the graph and update their feature representations; repeat the above process for several layers, and share the parameters of each layer; S3.

10. After multiple layers of processing, each node has an extracted feature vector.

5. The flexible job shop scheduling method based on deep reinforcement learning algorithm according to claim 1, characterized in that: The steps of the S5 network model training deep reinforcement learning method are as follows: The new feature matrix is ​​used as input, and the proximal optimization strategy is used to achieve decision-making; the Sctor network, the Critic network and the experience replay pool are connected in sequence to construct a neural network model for deep reinforcement learning; wherein the Sctor network and the Critic network are composed of four fully connected layers with sizes of 256, 256, 256 and 64 respectively; S5.

1. Input the feature vector into the Sctor network to obtain the action probability π(a t |s t ); MLP is a multi-layer perceptron that maps input to a scalar value. The state is s t The features extracted from the kth layer of GIN under ′(K) (s t ) is the state s t The features extracted from the kth layer of HGCN below are used to select actions from the action space using a greedy strategy; S5.

2. Execute the selected action a t , get the next moment state information S t+1 , and obtain immediate reward r through environmental feedback t And the state of the next moment S_t+1, (S t ,a t ,r t ,S t+1 ) is stored as experience in the experience replay pool; S5.2.1, the reward is estimated by the difference in expected completion time after executing action a_t, that is, S5.

3. Extract 128 experiences from the experience replay pool to update the neural network model of deep reinforcement learning; S5.3.

1. Calculate TD target: Among them, γ0 is the initial discount factor, t is the current time step, T is the total number of time steps, r t+k is the reward at the kth time step, s t+n is the state at time step t+n, π target is the target strategy, Q target is the output of the target Critic network, γ t+k is the dynamic discount factor at time step t+k; S5.3.

2. Update the Sctor network parameters. The loss of the Sctor network is: S5.3.

3. Update the Critic network parameters. The loss of the Critic network is: S5.4, repeat S5.1-S5.3 until all workpieces are scheduled; S5.

5. Extract 128 experiences from the experience replay pool to update the neural network model of deep reinforcement learning; S5.

6. Repeat S5.1-S5.5 with the updated deep reinforcement learning neural network until the maximum number of training rounds is reached, thereby completing the training of the deep reinforcement learning neural network model.

Images