A method for solving dynamic FJSP based on DIFFormer and deep reinforcement learning

Abstract

The application discloses a method for solving dynamic FJSP based on DIFFormer and deep reinforcement learning, and steps of the method are as follows: S1, constructing a D-DRL model for solving a dynamic flexible job shop scheduling problem; S2, defining the flexible job shop scheduling problem as a five-tuple heterogeneous graph; S2, converting the FJSP into a Markov decision process; S3, extracting features of the five-tuple heterogeneous graph by using the DIFFormer; S4, training a policy by using a double-delay deep deterministic policy gradient algorithm; S5, in the process of executing the flexible job shop scheduling scheme, for a dynamic event, according to a current execution progress and a proportion of the total scheduling, a suitable rescheduling strategy is used to modify the dynamic flexible job shop scheduling scheme; in conclusion, the method combines deep reinforcement learning with the DIFFormer, fully utilizes global information among tasks, and accurately models the correlation of tasks, so that the accuracy and efficiency of scheduling decisions are improved.

Description

Technical Field

[0001] This invention relates to the field of flexible job shop scheduling technology, and in particular to a method for solving dynamic FJSP based on DIFFormer and deep reinforcement learning. Background Technology

[0002] Deep reinforcement learning for solving the Flexible Job Shop Scheduling (FJSP) problem suffers from two main drawbacks. First, the training process is typically nondeterministic, easily influenced by initialization conditions, hyperparameter selection, and sample distribution. In FJSP, this training instability can lead to convergence failure or failure to achieve desired performance levels, meaning that a single training iteration may not yield a satisfactory scheduling strategy. To address this, multiple iterations and trials may be necessary, involving adjustments to initialization conditions, hyperparameter selection, the use of different training algorithms, or increases in training data, to find a more stable and high-performance scheduling scheme.

[0003] Secondly, deep reinforcement learning relies on the training and inference of large-scale neural networks, leading to high computational costs. Flexible job shop scheduling problems typically have large state and action spaces, meaning that the training and inference processes require significant computational resources and time. This may limit the feasibility of deep reinforcement learning in practical applications, especially where efficient real-time scheduling decisions are required.

[0004] The invention disclosed in CN116644902A presents a method for multi-objective dynamic flexible job shop scheduling based on deep reinforcement learning, which addresses energy consumption. The method includes the following steps: constructing a high-level deep reinforcement learning network and a low-level deep reinforcement learning network; obtaining the state features at the current scheduling moment and inputting them into the high-level deep reinforcement learning network for training; inputting the high-level optimization objective and state features into the low-level network and optimizing the network parameters; and selecting suitable workpieces and processing machines based on the obtained optimization objective values ​​to achieve the final scheduling optimization objective. However, this invention, when using deep reinforcement learning to solve the flexible job shop scheduling problem, is easily affected by initialization conditions, hyperparameter selection, and sample distribution, leading to training failures or difficulty in achieving ideal performance levels. Furthermore, the deep reinforcement learning in this invention relies on the training and inference of large-scale neural networks, resulting in high computational costs. Summary of the Invention

[0005] This invention addresses the problems of short-sightedness, excessive computation time, difficulty in determining algorithm parameters, and inability to fully utilize global information in traditional scheduling methods for solving the dynamic flexible job shop scheduling problem. It provides a method for solving dynamic FJSP based on DIFFormer and deep reinforcement learning.

[0006] The present invention adopts the following specific technical solutions:

[0007] A method for solving dynamic FJSPs based on DIFFormer and deep reinforcement learning, the steps of which are as follows:

[0008] S1: Construct a D-DRL model for solving the dynamic flexible job shop scheduling problem. The D-DRL model mainly includes a quintuple heterogeneous graph, a Markov decision model, a DIFFormer encoder, and a dual-delay deep deterministic policy gradient algorithm. The steps for using the D-DRL model to solve the dynamic flexible job shop scheduling problem are as follows:

[0009] S1.1: Construct a quintuple heterogeneous graph consisting of process nodes, machine nodes, and virtual nodes based on an example of the flexible work shop scheduling problem;

[0010] S1.2: Model the flexible job shop scheduling problem as a Markov decision process;

[0011] S1.3: Use the DIFFormer encoder to extract features from the quintuple heterogeneous graph constructed in step S1;

[0012] S1.4: The scheduling strategy model is trained using a dual-delay deep deterministic policy gradient algorithm;

[0013] S2: Read the instance of the flexible job shop scheduling problem, load the trained D-DRL model to generate an initial scheduling scheme;

[0014] S3: Execute the initial scheduling plan. During the scheduling process, when dynamic events such as machine failure, emergency order insertion, changes in the workpiece's process sequence, or changes in the process processing time occur, an appropriate rescheduling strategy is adopted. The calculation formula for the rescheduling strategy is as follows:

[0015]

[0016] Where D is the number of processes currently scheduled, and P i Let P be the number of processes for the i-th workpiece, β (0≤β≤1) be the proportion coefficient, and P be the number of processes for the i-th workpiece. CRS For a complete rescheduling strategy, P RSRS The strategy is to reschedule to the right.

[0017] S4: Repeat step S3 until the scheduling of the dynamic flexible job shop scheduling problem instance is completed.

[0018] Furthermore, the virtual node in step S1.1 includes a start node and an end node.

[0019] Furthermore, the steps to model the flexible job shop scheduling problem as a Markov decision process are as follows:

[0020] S1.2.1: Design the state space s consisting of process node features and machine node features. t The process node characteristics are determined by PM(O) ij ), PT(O ij ), SS(O ij ), PO(O ij ), SO(O ij ), ST(O ij ) and ET(O ij It consists of PM(O) ij ) represents the machine tool capable of performing this process, PT(O) ij ) represents the processing time, SS(O) ij ) represents the scheduling status of the process, PO(O) ij ) represents the preceding process on the same workpiece, SO(O) ij ST(O) represents the next operation on the same workpiece. ij ) represents the start time of this process, ET(O) ij The end time of this process is ); the machine node characteristics are determined by MS. t PS t and SP t MS t Indicates the status of the machine, PS t SP indicates the process that has been scheduled on the machine. t This represents the start time of the corresponding process.

[0021] S1.2.2: Construction Action in, This indicates that a process step has been selected. This indicates that you are selecting one machine.

[0022] S1.2.3: Calculate the average utilization rate of all machines. The maximum completion time is then normalized using decimal scaling.

[0023] Average utilization rate of all its machines The calculation formula is:

[0024]

[0025] In the above formula, k is the machine number, M is the total number of machines, and the machine utilization rate of machine Mk is... The formula is:

[0026]

[0027] in, This represents the processing time of the d-th operation already processed on machine tool Mk. After scheduling t steps, d′ corresponds to the last operation processed on the corresponding machine tool.

[0028] The formula for calculating the maximum completion time using decimal scaling and normalization is as follows:

[0029]

[0030] Where x is the value that makes |C′ t The smallest integer less than 1, C t For the maximum completion time;

[0031] S1.2.4: The reward function is designed as follows: Where λ and ε are weighting coefficients.

[0032] Furthermore, the process of extracting features from the quintuple heterogeneous graph constructed in step S1 using the DIFFormer encoder in step S1.3 is as follows:

[0033] S1.3.1: The first linear layer in the DIFFormer encoder performs a linear transformation on the node features of the process node and the node features of the machine node;

[0034] S1.3.2: Store the node features of the currently output process node and the node features of the machine node as residual links, and map the node features of the input process node and the node features of the machine node to the representation space of Key, Query and Value;

[0035] S1.3.3: Perform full attention convolution operation, and then perform graph convolution operation on this basis;

[0036] S1.3.4: Pool the result of graph convolution and output the processed node features as the state space.

[0037] Furthermore, the steps in step S1.4 for training the scheduling policy model using the dual-delay deep deterministic policy gradient algorithm are as follows:

[0038] S1.4.1: Construct a scheduling strategy model, which mainly includes an Actor network μ, a Critic1 network Q1, a Critic2 network Q2, a Target Actor network u′, a Target Critic1 network Q′1, and a Target Critic2 network Q′2. In all networks of the scheduling strategy model, there is one input layer, two hidden layers, and one output layer. The ReLU activation function is used in the hidden layers to improve the expressive power of the model.

[0039] S1.4.2: Initialize the network and experience buffer in the scheduling strategy model;

[0040] S1.4.3: Obtain the currently processable process nodes, use the Actor network to output an action based on the current state, the environment executes the action to select a pair of process-machine, and gives the next state and reward;

[0041] S1.4.4: Set the current state s t Current action a t Next state s t+1 The reward r is stored in the experience pool;

[0042] S1.4.5: In the Target Actor network, output the target action a based on the next state. t+1 and for a t+1 The formula for adding noise is as follows:

[0043] a t+1 =u′(s t+1 |θ u′ )+ξ

[0044] The values ​​y1 and y2 of the next state are calculated in the Target Critic network Q′1 and Q′2 respectively. The formulas for calculating y1 and y2 are shown below:

[0045]

[0046]

[0047] Where r is the reward, γ is the discount factor, and s t+1 For the next state, θ u′ Let μ be the parameter of the TargetActor network, and select the minimum of the two values ​​as the target Q value for subsequent training;

[0048] S1.4.6: When the experience pool reaches a certain size, a batch of samples is sampled from it to update the network parameters. The gradient descent method is used to update the network parameters of Critic1 and Critic2. The formula for calculating the loss function used to update the network parameters is as follows:

[0049]

[0050] Among them, L C Loss function, B is the number of samples, y b For the target value of the b-th sample, Q u (s b a b ) is state s band action a b The predicted value;

[0051] The Actor network parameters are updated using a deterministic policy gradient at a frequency of delay update steps d. The update formula for the Actor network parameters is as follows:

[0052]

[0053] in, Represents θ u The gradient in state s;

[0054] S1.4.7: Update the parameters of the TargetActor network u′ using a soft update method. The formula for updating the parameters of the TargetActor network u′ is as follows:

[0055] θ u′ =σθ u +(1-σ)θ u′

[0056] Where σ is the soft update coefficient;

[0057] S1.4.8: Update the parameters of Target Critic1 network Q′1 and Target Critic2 network Q′2 using a soft update method, with the following formula:

[0058]

[0059] S1.4.9: Repeat steps S1.4.3-S1.4.9, and after multiple rounds of iterative training, output the trained scheduling strategy model.

[0060] Furthermore, in step S3, according to the calculation formula of the rescheduling strategy, if it is determined whether the current progress exceeds half of the total progress, the right-shift rescheduling strategy is adopted to modify the scheduling scheme; otherwise, the full rescheduling strategy is adopted to modify the scheduling scheme.

[0061] Furthermore, the method is verified and analyzed through experiments.

[0062] The beneficial effects of this invention are as follows:

[0063] The beneficial effects of the method for solving dynamic FJSP based on DIFFormer and deep reinforcement learning designed in this invention are as follows:

[0064] (1) The method of the present invention uses DIFFormer to extract features, which can make full use of global information between tasks and accurately model the correlation between tasks, thereby improving the accuracy and efficiency of scheduling decisions.

[0065] (2) This invention combines deep reinforcement learning with DIFFormer. Specifically, the TD3 algorithm is used to train the scheduling policy. The TD3 algorithm uses an experience pool to store training samples, from which random samples are taken for training, increasing sample utilization, improving training efficiency and stability, and reducing sample correlation problems during training. Secondly, the method of this invention alleviates instability during training by delaying the updates of the target policy network and the Q-network, improving the convergence and stability of the algorithm. Simultaneously, a dual Q-network is used to estimate the action value function. Two independent Q-networks mitigate the problem of overestimating the action value function, thereby improving the accuracy of the algorithm. Attached Figure Description

[0066] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0067] Figure 1 This is a schematic diagram of the D-DRL model framework;

[0068] Figure 2 A simplified flowchart of a method for solving dynamic FJSPs using a D-DRL model based on DIFFormer and deep reinforcement learning;

[0069] Figure 3 A schematic diagram of the DIFFormer encoder structure used for feature extraction;

[0070] Figure 4 A schematic diagram of the scheduling strategy training for the TD3 algorithm. Detailed Implementation

[0071] The following examples further explain and clarify the invention, but the specific examples do not limit the invention in any way.

[0072] Example 1

[0073] A method for solving dynamic FJSPs based on DIFFormer and deep reinforcement learning, such as Figures 1-4 As shown, the steps of the method are as follows:

[0074] S1: Construct a D-DRL model for solving the dynamic flexible job shop scheduling problem. The D-DRL model mainly includes a quintuple heterogeneous graph, a Markov decision model, a DIFFormer encoder, and a dual-delay deep deterministic policy gradient algorithm. The steps for using the D-DRL model to solve the dynamic flexible job shop scheduling problem are as follows:

[0075] S1.1: Based on the example of the flexible work shop scheduling problem, construct a quintuple heterogeneous graph consisting of process nodes, machine nodes, and virtual nodes, where the virtual nodes include start nodes and end nodes;

[0076] S1.2: Model the flexible job shop scheduling problem as a Markov decision process;

[0077] S1.3: Use the DIFFormer encoder to extract features from the quintuple heterogeneous graph constructed in step S1;

[0078] S1.4: The scheduling strategy model is trained using a dual-delay deep deterministic policy gradient algorithm;

[0079] S2: Read the instance of the flexible job shop scheduling problem, load the trained D-DRL model to generate an initial scheduling scheme;

[0080] S3: Execute the initial scheduling plan. During the scheduling process, when dynamic events occur, an appropriate rescheduling strategy is adopted. The calculation formula for the rescheduling strategy is as follows:

[0081]

[0082] Where D is the number of processes currently scheduled, and P i Let P be the number of processes for the i-th workpiece, β (0≤β≤1) be the proportion coefficient, and P be the number of processes for the i-th workpiece. CRS For a complete rescheduling strategy, P RSRS The strategy is to reschedule to the right.

[0083] According to the calculation formula of the rescheduling strategy, if it is determined whether the current progress exceeds half of the total progress, the right-shift rescheduling strategy is used to modify the scheduling scheme; otherwise, the full rescheduling strategy is used to modify the scheduling scheme.

[0084] S4: Repeat step S3 until the scheduling of the dynamic flexible job shop scheduling problem instance is completed.

[0085] Furthermore, the steps to model the flexible job shop scheduling problem as a Markov decision process are as follows:

[0086] S1.2.1: Design the state space s consisting of process node features and machine node features. t The process node characteristics are determined by PM(O) ij ), PT(O ij ), SS(O ij ), PO(O ij ), SO(O ij ), ST(O ij ) and ET(Oij Composed of MS, the machine node features are characterized by MS. t PS t and SP t ;

[0087] S1.2.2: Construction Action in, This indicates that a process step has been selected. This indicates that you are selecting one machine.

[0088] S1.2.3: Calculate the average utilization rate of all machines. The maximum completion time is then normalized using decimal scaling.

[0089] Average utilization rate of all its machines The calculation formula is:

[0090]

[0091] In the above formula, k is the machine number, M is the total number of machines, and the machine utilization rate of machine Mk is... The formula is:

[0092]

[0093] in, This represents the processing time of the d-th operation already processed on machine tool Mk. After scheduling t steps, d′ corresponds to the last operation processed on the corresponding machine tool.

[0094] The formula for calculating the maximum completion time using decimal scaling and normalization is as follows:

[0095]

[0096] Where x is the value that makes |C′ t The smallest integer less than 1, C t For the maximum completion time;

[0097] S1.2.4: The reward function is designed as follows: Where λ and ε are weighting coefficients.

[0098] Furthermore, the steps in step S1.4 for training the scheduling policy model using the dual-delay deep deterministic policy gradient algorithm are as follows:

[0099] S1.4.1: Construct a scheduling strategy model, which mainly includes an Actor network μ, a Critic1 network Q1, a Critic2 network Q2, a Target Actor network u′, a Target Critic1 network Q′1, and a Target Critic2 network Q′2. In all networks of the scheduling strategy model, there is one input layer, two hidden layers, and one output layer. The ReLU activation function is used in the hidden layers to improve the expressive power of the model.

[0100] S1.4.2: Initialize the network and experience buffer in the scheduling strategy model;

[0101] S1.4.3: Obtain the currently processable process nodes, use the Actor network to output an action based on the current state, the environment executes the action to select a pair of process-machine, and gives the next state and reward;

[0102] S1.4.4: Set the current state s t Current action a t Next state s t+1 The reward r is stored in the experience pool;

[0103] S1.4.5: In the Target Actor network, output the target action a based on the next state. t+1 and for a t+1 The formula for adding noise is as follows:

[0104] a t+1 =u′(s t+1 |θ u′ )+ξ

[0105] The values ​​y1 and y2 of the next state are calculated in the Target Critic network Q′1 and Q′2 respectively. The formulas for calculating y1 and y2 are shown below:

[0106]

[0107]

[0108] Where r is the reward, γ is the discount factor, and s t+1 For the next state, θ u′ For the parameters of the TargetActor network μ′, select the minimum of the two values ​​as the target Q value for subsequent training;

[0109] S1.4.6: When the experience pool reaches a certain size, a batch of samples is sampled from it to update the network parameters. The gradient descent method is used to update the network parameters of Critic1 and Critic2. The formula for calculating the loss function used to update the network parameters is as follows:

[0110]

[0111] Among them, L C Loss function, B is the number of samples, y b For the target value of the b-th sample, Q u (s b a b ) is state s b and action a b The predicted value;

[0112] The Actor network parameters are updated using a deterministic policy gradient at a frequency of delay update steps d. The update formula for the Actor network parameters is as follows:

[0113]

[0114] S1.4.7: Update the parameters of the Target Actor network u′ using a soft update method. The formula for updating the parameters of the Target Actor network u′ is as follows:

[0115] θ u′ =σθ u +(1-σ)θ u′

[0116] S1.4.8: Update the parameters of Target Critic1 network Q′1 and Target Critic2 network Q′2 using a soft update method, with the following formula:

[0117]

[0118] S1.4.9: Repeat steps S1.4.3-S1.4.9, and after multiple rounds of iterative training, output the trained scheduling strategy model.

[0119] Furthermore, the method is verified and analyzed through experiments.

[0120] The beneficial effects of this invention are as follows:

[0121] In summary, this invention combines deep reinforcement learning with DIFFormer. Specifically, the TD3 algorithm is used to train the scheduling policy. The TD3 algorithm uses an experience pool to store training samples, from which random sampling is performed for training, increasing sample utilization, improving training efficiency and stability, and reducing sample correlation issues during training. Secondly, by delaying the updates of the target policy network and the Q-network, instability during training can be mitigated, improving the algorithm's convergence and stability. Simultaneously, a dual Q-network is employed to estimate the action value function. Two independent Q-networks mitigate the problem of overestimating the action value function, thereby improving the algorithm's accuracy.

[0122] Example 2

[0123] A method for solving dynamic FJSPs based on DIFFormer and deep reinforcement learning, such as Figures 1-4 As shown, the steps of the method are as follows:

[0124] S1: Construct a D-DRL model for solving the dynamic flexible job shop scheduling problem. The D-DRL model mainly includes a quintuple heterogeneous graph, a Markov decision model, a DIFFormer encoder, and a dual-delay deep deterministic policy gradient algorithm. The steps for using the D-DRL model to solve the dynamic flexible job shop scheduling problem are as follows:

[0125] S1.1: Based on the example of the flexible work shop scheduling problem, construct a quintuple heterogeneous graph consisting of process nodes, machine nodes, and virtual nodes, where the virtual nodes include start nodes and end nodes;

[0126] S1.2: Model the flexible job shop scheduling problem as a Markov decision process;

[0127] S1.3: Use the DIFFormer encoder to extract features from the quintuple heterogeneous graph constructed in step S1;

[0128] The process of feature extraction from the quintuple heterogeneous graph constructed in step S1 using the DIFFormer encoder is as follows:

[0129] S1.3.1: The first linear layer in the DIFFormer encoder performs a linear transformation on the node features of the process node and the node features of the machine node;

[0130] S1.3.2: Store the node features of the currently output process node and the node features of the machine node as residual links, and map the node features of the input process node and the node features of the machine node to the representation space of Key, Query and Value;

[0131] S1.3.3: Perform full attention convolution operation, and then perform graph convolution operation on this basis;

[0132] S1.3.4: Pool the result of graph convolution and output the processed node features as the state space;

[0133] S1.4: The scheduling strategy model is trained using a dual-delay deep deterministic policy gradient algorithm;

[0134] S2: Read the instance of the flexible job shop scheduling problem, load the trained D-DRL model to generate an initial scheduling scheme;

[0135] S3: Execute the initial scheduling plan. During the scheduling process, when dynamic events occur, an appropriate rescheduling strategy is adopted. The calculation formula for the rescheduling strategy is as follows:

[0136]

[0137] Where D is the number of processes currently scheduled, and P i Let P be the number of processes for the i-th workpiece, β (0≤β≤1) be the proportion coefficient, and P be the number of processes for the i-th workpiece. CRS For a complete rescheduling strategy, P RSRS The strategy is to reschedule to the right.

[0138] According to the calculation formula of the rescheduling strategy, if it is determined whether the current progress exceeds half of the total progress, the right-shift rescheduling strategy is used to modify the scheduling scheme; otherwise, the full rescheduling strategy is used to modify the scheduling scheme.

[0139] S4: Repeat step S3 until the scheduling of the dynamic flexible job shop scheduling problem instance is completed.

[0140] Furthermore, the steps to model the flexible job shop scheduling problem as a Markov decision process are as follows:

[0141] S1.2.1: Design the state space s consisting of process node features and machine node features. t The process node characteristics are determined by PM(O) ij ), PT(O ij ), SS(O ij ), PO(O ij ), SO(O ij ), ST(O ij ) and ET(O ij Composed of MS, the machine node features are characterized by MS. t PS t and SP t ;

[0142] S1.2.2: Construction Action in, This indicates that a process step has been selected. This indicates that you are selecting one machine.

[0143] S1.2.3: Calculate the average utilization rate of all machines. The maximum completion time is then normalized using decimal scaling.

[0144] Average utilization rate of all its machines The calculation formula is:

[0145]

[0146] In the above formula, k is the machine number, M is the total number of machines, and the machine utilization rate of machine Mk is... The formula is:

[0147]

[0148] in, This represents the processing time of the d-th operation already processed on machine tool Mk. After scheduling t steps, d′ corresponds to the last operation processed on the corresponding machine tool.

[0149] The formula for calculating the maximum completion time using decimal scaling and normalization is as follows:

[0150]

[0151] Where x is the value that makes |C′ t The smallest integer less than 1, C t For the maximum completion time;

[0152] S1.2.4: The reward function is designed as follows: Where λ and ε are weighting coefficients.

[0153] Furthermore, the steps in step S1.4 for training the scheduling policy model using the dual-delay deep deterministic policy gradient algorithm are as follows:

[0154] S1.4.1: Construct a scheduling strategy model, which mainly includes an Actor network μ, a Critic1 network Q1, a Critic2 network Q2, a Target Actor network u′, a Target Critic1 network Q′1, and a Target Critic2 network Q′2. In all networks of the scheduling strategy model, there is one input layer, two hidden layers, and one output layer. The ReLU activation function is used in the hidden layers to improve the expressive power of the model.

[0155] S1.4.2: Initialize the network and experience buffer in the scheduling strategy model;

[0156] S1.4.3: Obtain the currently processable process nodes, use the Actor network to output an action based on the current state, the environment executes the action to select a pair of process-machine, and gives the next state and reward;

[0157] S1.4.4: Set the current state s t Current action a t Next state s t+1 The reward r is stored in the experience pool;

[0158] S1.4.5: In the Target Actor network, output the target action a based on the next state. t+1 and for a t+1 The formula for adding noise is as follows:

[0159] a t+1 =u′(s t+1 |θ u′ )+ξ

[0160] The values ​​y1 and y2 of the next state are calculated in the Target Critic network Q′1 and Q′2 respectively. The formulas for calculating y1 and y2 are shown below:

[0161]

[0162]

[0163] Where is the reward, is the discount factor, is the next state, and is the parameter of the Target Actor network. The minimum of the two values ​​is selected as the target Q value for subsequent training.

[0164] S1.4.6: When the experience pool reaches a certain size, a batch of samples is sampled from it to update the network parameters. The gradient descent method is used to update the network parameters of Critic1 and Critic2. The formula for calculating the loss function used to update the network parameters is as follows:

[0165]

[0166] Among them, L C Loss function, B is the number of samples, y b For the target value of the b-th sample, Q u (s b a b ) is state s b and action a b The predicted value;

[0167] The Actor network parameters are updated using a deterministic policy gradient at a frequency of delay update steps d. The update formula for the Actor network parameters is as follows:

[0168]

[0169] S1.4.7: Update the parameters of the Target Actor network u′ using a soft update method. The formula for updating the parameters of the Target Actor network u′ is as follows:

[0170] θ u′ =σθ u +(1-σ)θ u′

[0171] S1.4.8: Update the parameters of Target Critic1 network Q′1 and Target Critic2 network Q′2 using a soft update method, with the following formula:

[0172]

[0173] S1.4.9: Repeat steps S1.4.3-S1.4.9, and after multiple rounds of iterative training, output the trained scheduling strategy model.

[0174] Furthermore, the method is verified and analyzed through experiments.

[0175] In summary, this invention employs DIFFormer to extract features, which can fully utilize global information between tasks and accurately model task correlations, thereby improving the accuracy and efficiency of scheduling decisions. Furthermore, this invention combines deep reinforcement learning with DIFFormer, using the TD3 algorithm to train the scheduling policy, increasing sample utilization, improving training efficiency and stability, and reducing sample correlation issues during training. Secondly, by delaying the updates of the target policy network and Q-network, instability during training can be mitigated, improving the algorithm's convergence and stability.

[0176] Example 3

[0177] As shown in Table 1, the execution time (in seconds) and maximum completion time of FJSP instances in the Brandimarte benchmark dataset solved by the heuristic scheduling rules and the method proposed in this invention are as follows: t represents the execution time, and ms represents the maximum completion time. Rule 1 prioritizes the operation with the smallest priority operation number, Rule 2 prioritizes the operation with the shortest processing time, and Rule 3 prioritizes the operation with the longest remaining processing time. Experimental results show that the heuristic scheduling rules resulted in a minimum execution time of 0.19 seconds and a maximum of 0.23 seconds for the MK01 instance, while the D-DRL execution time was 0.21 seconds. Similarly, the heuristic scheduling rules resulted in a minimum execution time of 0.47 seconds and a maximum of 0.52 seconds for the MK05 instance, while the D-DRL execution time was 0.51 seconds. The optimal heuristic scheduling rule for MK03 is rule 3, with a result of 212, while the D-DRL solution yields 204. The optimal heuristic scheduling rule for MK04 is rule 1, with a result of 75, while the D-DRL solution yields 62. Therefore, the heuristic scheduling rule and the method proposed in this invention are comparable in execution time. However, in terms of solution quality, the maximum completion time obtained by the method proposed in this invention is superior to that of the heuristic scheduling rule.

[0178]

[0179] Table 1

[0180] As shown in Table 2, the maximum completion time for solving the FJSP instance with random dynamic events is achieved using the full rescheduling strategy, the rescheduling strategy, and the selective rescheduling strategy proposed in this invention. The experimental results show that when solving D_MK01, the maximum completion time using the full rescheduling strategy is 54, the right-shift rescheduling strategy is 60, while the maximum completion time using the selective rescheduling strategy proposed in this invention is 48. Similarly, when solving D_MK02, D_MK03, D_MK04, and D_MK05, the maximum completion time using the selective rescheduling strategy is shorter than that obtained by the full rescheduling strategy and the right-shift rescheduling strategy. Therefore, the maximum completion time of the dynamic flexible job shop scheduling problem instance solved by the selective rescheduling strategy proposed in this invention is superior to that of full rescheduling and right-shift rescheduling.

[0181] D_MK01 54 60 48 D_MK02 43 47 36 D_MK03 235 226 223 D_MK04 82 78 75 D_MK05 201 195 193

[0182] Table 2

[0183] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the present invention, and are not intended to limit the implementation of the present invention. Those skilled in the art can make other variations or modifications based on the above description. It is neither necessary nor possible to exhaustively describe all embodiments here. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for solving dynamic FJSPs based on DIFFormer and deep reinforcement learning, characterized in that, The steps of the method are as follows: S1: Construct a D-DRL model for solving the dynamic flexible job shop scheduling problem. The D-DRL model mainly includes a quintuple heterogeneous graph, a Markov decision model, a DIFFormer encoder, and a dual-delay deep deterministic policy gradient algorithm. The steps for using the D-DRL model to solve the dynamic flexible job shop scheduling problem are as follows: S1.1: Construct a quintuple heterogeneous graph consisting of process nodes, machine nodes, and virtual nodes based on an example of the flexible work shop scheduling problem; S1.2: Model the flexible job shop scheduling problem as a Markov decision process; S1.2.1: Design the state space consisting of process node features and machine node features. The process node features are composed of , , , , , and Composed of, the machine node features are composed of , and ; S1.2.2: Construction Action ,in, This indicates that a process step has been selected. This indicates that you are selecting one machine. S1.2.3: Calculate the average utilization rate of all machines. The maximum completion time is then normalized using decimal scaling. S1.2.4: The reward function is designed as follows: Where λ and ε are weighting coefficients. The normalized value of the maximum completion time is expressed as: Where x is such that The smallest integer, For the maximum completion time; S1.3: Use the DIFFormer encoder to extract features from the quintuple heterogeneous graph constructed in step S1; S1.3.1: The first linear layer in the DIFFormer encoder performs a linear transformation on the node features of the process node and the node features of the machine node; S1.3.2: Store the node features of the currently output process node and the node features of the machine node as residual links, and map the node features of the input process node and the node features of the machine node to the representation space of Key, Query and Value; S1.3.3: Perform full attention convolution operation, and then perform graph convolution operation on this basis; S1.3.4: Pool the result of graph convolution and output the processed node features as the state space; S1.4: The scheduling strategy model is trained using a dual-delay deep deterministic policy gradient algorithm; S2: Read the instance of the flexible job shop scheduling problem, load the trained D-DRL model to generate an initial scheduling scheme; S3: Execute the initial scheduling plan. During the scheduling process, when dynamic events occur, an appropriate rescheduling strategy is adopted. The calculation formula for the rescheduling strategy is as follows: in, D This represents the number of processes that have been scheduled. P i For the first i Number of processes per workpiece ( () represents the percentage coefficient. For a complete rescheduling strategy, The strategy is to reschedule to the right. S4: Repeat step S3 until the scheduling of the dynamic flexible job shop scheduling problem instance is completed.

2. The method for solving dynamic FJSPs based on DIFFormer and deep reinforcement learning according to claim 1, characterized in that, The virtual node includes a start node and an end node.

3. The method for solving dynamic FJSPs based on DIFFormer and deep reinforcement learning according to claim 1, characterized in that, The average utilization rate of all machines in step S1.2.3 The calculation formula is: = In the above formula, Machine number The machine utilization rate of machine Mk is the total number of machines. The formula is: = in, This indicates the processing time of the d-th operation already completed on machine tool Mk, after scheduling t steps. This corresponds to the final processing step on the machine.

4. The method for solving dynamic FJSPs based on DIFFormer and deep reinforcement learning according to claim 1, characterized in that, The steps in step S1.4 for training the scheduling strategy model using the dual-delay deep deterministic strategy gradient algorithm are as follows: S1.4.1: Construct a scheduling strategy model, which mainly includes the Actor network μ and the Critic1 network. Critic2 network Target Actor Network Target Critic1 network and the Target Critic2 network ; In all networks of the scheduling strategy model, there is one input layer, two hidden layers and one output layer. The ReLU activation function is used in the hidden layer to improve the expressive power of the model. S1.4.2: Initialize the network and experience buffer in the scheduling strategy model; S1.4.3: Obtain the currently processable process nodes, use the Actor network to output an action based on the current state, the environment executes the action to select a pair of process-machine, and gives the next state and reward; S1.4.4: Set the current state Current action Next state and rewards Stored in the experience pool; S1.4.5: In the Target Actor network, output the target action based on the next state. and to Add noise to the Target Critic network. and The value of the next state is calculated separately. The minimum of the two values ​​is selected as the target Q-value for subsequent training. The calculation formulas are as follows: ， in, As a reward, As a discount factor, For the next state, For Target Actor Network Parameters; S1.4.6: When the experience pool reaches a certain size, a batch of samples is sampled from it to update the network parameters. The gradient descent method is used to update the network parameters of Critic1 and Critic2, and then a deterministic gradient descent is used to update the Actor network parameters with a delayed update step. d The frequency of updates is as follows, and the update formula for the Actor network parameters is as follows: in, express In state The gradient below; S1.4.7: Soft update the Target Actor network Update the parameters; S1.4.8: Update the Target Critic1 network using a soft update method. Target Critic2 network The formula for updating parameters is as follows: S1.4.9: Repeat steps S1.4.3-S1.4.9, and after multiple rounds of iterative training, output the trained scheduling strategy model.

5. The method for solving dynamic FJSPs based on DIFFormer and deep reinforcement learning according to claim 4, characterized in that, In step S1.4.6, the network parameters of Critic1 and Critic2 are updated using a loss function, the calculation formula of which is as follows: in, loss function It is the sample size. Is for the first b The target value for each sample It is a state and actions The predicted value.

6. The method for solving dynamic FJSPs based on DIFFormer and deep reinforcement learning according to claim 1, characterized in that, In step S3, according to the calculation formula of the rescheduling strategy, if it is determined whether the current progress exceeds half of the total progress, the right-shift rescheduling strategy is adopted to modify the scheduling scheme; otherwise, the full rescheduling strategy is adopted to modify the scheduling scheme.

7. The method for solving dynamic FJSPs based on DIFFormer and deep reinforcement learning according to claim 1, characterized in that, The method was verified and analyzed experimentally.

Images