Reinforcement learning and graph neural fusion algorithm for effectively solving flexible workshop scheduling

Abstract

The invention discloses a reinforcement learning and graph neural fusion algorithm for effectively solving flexible workshop scheduling, which comprises the following steps of: formalizing an FJSP into an MDP by setting states, actions, state conversion and rewards, and describing a scheduling process as dynamically allocating ready operation to a compatible and idle machine; in a scheduling process, firstly converting a scheduling state into a heterogeneous graph structure; carrying out two-stage embedding by adopting a heterogeneous graph attention network, and extracting features of jobs and machines from a heterogeneous graph for embedding; the decision network generates action probability distribution by using feature embedding, and samples scheduling actions from the action probability distribution; and training the deep reinforcement learning model by using a near-end strategy optimization algorithm. Compared with the traditional PDR and SOTA DRL methods, the algorithm has the advantages that a high-quality solution is obviously realized on two synthetic data sets with different distributions, and the performance and generalization ability are improved. Good generalization ability is also exhibited because the model trains large-scale or out-of-distribution instances on small-scale instances.

Description

Technical Field

[0001] The present invention relates to the technical field of flexible workshop scheduling, and in particular to a reinforcement learning and graph neural fusion algorithm for effectively solving flexible workshop scheduling. Background Art

[0002] The flexible job shop scheduling problem (FJSP) is widely used in many fields such as semiconductor manufacturing, automobile production and textile to maintain production efficiency and cost-effectiveness. With the rapid development of information technology, FJSP has gradually become a hot issue for flexible manufacturing companies because it is more flexible and complex than the job shop scheduling problem (JSSP) that relaxes machine constraints, allowing each operation to be processed on multiple compatible machines.

[0003] In recent years, deep reinforcement learning (DRL) has become a new research hotspot for JSP because of its ability to respond quickly to dynamic environments and make online decisions. Reinforcement learning-based scheduling agents can interact with their environment and automatically learn optimal policies, which are suitable for solving the stochastic dynamic multi-stage sequential decision-making problem of JSP. Typically, parameterized neural network models accept information from the production environment as input and output probability distributions of feasible scheduling actions, forming an end-to-end learning approach. By training on production process data and modeling the scheduling process as a Markov decision process (MDP), the DRL model can adaptively select the optimal action based on the current state to maximize the total reward associated with the FJSP production target. The introduced disjunctive graph for job shop scheduling (JSP) effectively represents the flexible job shop scheduling problem (FJSP) instance, representing the sequence constraints between jobs and the situation where multiple operations are processed on the same machine. After modeling the scheduling problem as a disjunctive graph, recent studies have shown that graph neural networks (GNNs) can effectively find solutions under complex constraints. Although the relationship between operation nodes is modeled by a disjunctive graph, the relationship between machines and operations has not yet been modeled in the graph to fully exploit the complex characteristics of FJSP. Nowadays, heterogeneous graphs are introduced to model the relationship between operations and machines by appending a set of machine nodes to the original disjoint graph. To further model the competition between machines for unscheduled operations, a graph attention network (GAT) is introduced to learn the embeddings of machines and operations separately;

[0004] Faced with complex FJSPs of various scales, effectively solving them using deep reinforcement learning (DRL) methods still faces three major challenges: (1) how to reasonably model the competitive relationship between compatible machines; (2) how to extract global and local scheduling information through deep neural networks to obtain more accurate state representation; and (3) how to capture dynamically evolving scheduling characteristics to effectively solve FJSPs of different scales. Summary of the invention

[0005] The purpose of the present invention is to provide an effective reinforcement learning and graph neural fusion algorithm for solving flexible shop scheduling in response to the above-mentioned problems. Operations and machines are modeled from a global and local perspective through dual heterogeneous graphs, and state features are learned through an attention mechanism to better capture the dynamic features of graphs with powerful expressiveness.

[0006] The technical solution of the present invention is as follows:

[0007] An effective reinforcement learning and graph neural network fusion algorithm for solving flexible shop scheduling includes the following steps:

[0008] The FJSP is formalized as an MDP by setting states, actions, state transitions, and rewards, while the scheduling process is described as dynamically allocating ready operations to compatible and idle machines;

[0009] In the scheduling process, the scheduling state is first converted into a heterogeneous graph structure;

[0010] A two-stage embedding is performed using a heterogeneous graph attention network to extract job and machine feature embedding from heterogeneous graphs, including: taking node features as input and projecting features of different types of nodes into the same feature space through a transformation matrix; using an improved dynamic graph attention mechanism to learn the attention weights between nodes and neighbors;

[0011] The decision network uses feature embedding to generate action probability distribution, from which scheduling actions are sampled;

[0012] Use the proximal policy optimization algorithm to train deep reinforcement learning models.

[0013] Furthermore, the method of taking the node features as input and projecting the features of different types of nodes into the same feature space through a transformation matrix specifically includes the following steps:

[0014] For each type of node, the transformation matrix Project the features of different types of nodes into the same feature space, transform matrix Based on the node type, different feature spaces of various nodes are processed, and the projection process is:

[0015]

[0016] Among them, h i and h′ i are the original features and projection features of node i respectively, and generate a new set of node features:

[0017]

[0018] in, is the feature space after projecting h, F′h is the projection operation on h.

[0019] Furthermore, the use of the improved dynamic graph attention mechanism to learn the attention weights between nodes and neighbors specifically includes:

[0020] Instantiation is handled by computing the weighted average of node representations. DGAT is proposed by modifying the internal operation order in GAT and integrating GATv2 to transform node features and handle all neighbors of node j in heterogeneous graphs.

[0021] Applying an A layer after the non-linearity and a W layer after the concatenation effectively applies a multi-layer perceptron to compute the score for each query-key pair:

[0022] e(h i ,h j )=a T LeakyReLU(W·[h i ||h j ]),

[0023] Among them, LeakyReLU is nonlinear.

[0024] Furthermore, the decision network includes the following contents:

[0025] Use DRL to train the agent by combining actor-critic and gradient policy with value function approximation;

[0026] Using the state features extracted from GAT, two multilayer perceptrons are set up as actors and critics with different parameters;

[0027] The actor network generates a probability distribution of actions based on the input features and determines the action selection of the agent in the current state; the critic network estimates the value function of the current state, indicating the expected cumulative reward of this state;

[0028] During training, the parameters θ of the actor network are updated via the policy gradient method; the parameters φ of the critic network are updated by minimizing the estimated error of the value function.

[0029] Furthermore, the training of the deep reinforcement learning model using the proximal strategy optimization algorithm specifically includes:

[0030] Trained by using PPO-clip as a scheduling agent;

[0031] For each training set, the environment and state are reset at the beginning; the simulation runs until a decision point t appears, and the current state of the system and the object queue are input into the PPO actor network; based on the current policy π, the SoftMax probability distribution of actions is output; the agent randomly samples actions according to this distribution and returns them to the environment;

[0032] A scheduling action is selected and which job in the queue should be processed; the simulation continues until the next decision point t+1 and the information of all jobs is updated again; the reward time of the action is calculated based on the idle time and fed back to the PPO;

[0033] At each step, record the action probability p(a) obtained by the critic network t ) and the state value function V(s t ), the trajectory (s t ,a t ,r t ,V(s t ),p(a t )) is stored in the storage module;

[0034] After a fixed number of steps, the batch advantage function a is calculated using the generalized advantage estimate; the trajectory is divided into small batches, and the network parameters of the actor and critic are repeatedly updated using the data, and the new trajectory is used in the subsequent learning phase; the PPO scheduling agent uses the new strategy π new Interact with the environment.

[0035] Further, the FJSP is formalized into an MDP by setting states, actions, state transitions and rewards, including: at step t, the DRL model receives a state s from the environment t and take action a t , assigns compatible actions to machines for immediate processing, with processing time Ts(t), and the FJSP environment returns a reward r associated with the completion time t ,Repeat the process for all operations |O| with heavy tasks and obtain the solution of FJSP.

[0036] Furthermore, the setting of the MDP includes:

[0037] Status: Status t ∈S contains all operations and machine features related to the current decision of step t; where s0 is the initial state and each state is described by a heterogeneous graph;

[0038] Action: The set A(t) of all compatible action-machine pairs sets the action space for step t, specifically including: action a t ∈A(t) is set to the OM pair (O ij ,M k ), where Oij is qualified, that is, its direct potential operation has been completed, and M k ∈M ij for the idle;

[0039] State transfer: based on s t and a t , the environment updates the current operation and machine state to obtain the new state s t+1 ;

[0040] Reward: Reward r t The role of is to guide the agent to choose the operation that helps to minimize the maximum completion time of all operations C max action, at step t, estimate each operation O ij The completion time of the next term is C(O ij ,s t ), and C(O i0 )=0; if O ij Already dispatched, O ij The completion time is C ij ; For all unscheduled operations, the completion time of the next recursive approximation is:

[0041]

[0042] in, For operation O ij Processing time on machine k;

[0043] The reward r is the estimated total completion time difference between steps t and t+1. t The calculation formula is:

[0044]

[0045] Furthermore, the heterogeneous graph structure includes a heterogeneous disjunctive graph and a competitive dynamic graph. The heterogeneous disjunctive graph structure is H = (O, M, C, ε), where the node set is the set of all operations, O S and O T To handle the start and end virtual nodes, the set C contains directed arcs representing the precedence constraints between consecutive operations of the same job, and the set M is the machine nodes, each of which corresponds to machine M. k , the set s is the set of operation-machine arcs, where each element E ijk ∈ε is an undirected arc;

[0046] The structure of the competition dynamic graph is D=(O,M,E), where O is the set of current candidate operations, M is the set of candidate machines, and E is the set of arcs between operations and machines. In the competition dynamic graph, nodes represent operations and machines respectively, and edges represent the connections and competitions between them.

[0047] Compared with the prior art, the present invention has the following beneficial effects:

[0048] 1. An effective reinforcement learning and graph neural fusion algorithm for solving flexible shop scheduling. An innovative end-to-end DRL algorithm is proposed. Operations and machines are modeled from global and local perspectives through dual heterogeneous graphs. Specifically, the heterogeneous disjunctive graph describes the entire operation and machine nodes to capture global features, while the dynamic competition graph encodes the current candidate nodes and possible machines to capture local features.

[0049] 2. An effective reinforcement learning and graph neural fusion algorithm for solving flexible shop scheduling. Different from the traditional GAT, a new attention mechanism is proposed to learn state features to better capture the dynamic features of the graph with powerful expressiveness;

[0050] 3. An effective reinforcement learning and graph neural fusion algorithm for solving flexible shop scheduling. Compared with traditional PDR and SOTA DRL methods, the algorithm of this application significantly achieves high-quality solutions on two synthetic datasets with different distributions, improving performance and generalization capabilities. In addition, it also shows good generalization ability because the model trains large-scale or out-of-distribution instances on small-scale instances. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 This is the framework diagram of the algorithm of this application.

[0052] Figure 2 This is the heterogeneous disjunction graph of the algorithm of this application.

[0053] Figure 3 This is the dynamic competition graph of the algorithm in this application.

[0054] Figure 4 This is the network architecture diagram of the algorithm used in this application. DETAILED DESCRIPTION

[0055] It should be noted that relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0056] The features and performance of the present invention are further described in detail below in conjunction with the embodiments.

[0057] See also Figure 1-4 , an effective reinforcement learning and graph neural network fusion algorithm for solving flexible shop scheduling, such as Figure 1 The following steps are included:

[0058] The FJSP is formalized as a Markov decision process (MDP) by setting states, actions, state transitions, and rewards, while the scheduling process is described as dynamically allocating ready operations to compatible and idle machines;

[0059] The FJSP is formalized as an MDP by setting states, actions, state transitions, and rewards. At step t, the DRL model receives the state s from the environment. t and take action a t , assigns compatible actions to machines for immediate processing, with processing time Ts(t), and the FJSP environment returns a reward r associated with the completion time t ,Repeat the process for all operations |O| with heavy tasks and obtain the solution of FJSP.

[0060] The MDP settings include:

[0061] Status: Status t ∈S contains all operations and machine features related to the current decision of step t; where s0 is the initial state and each state is described by a heterogeneous graph; since completed operations and machines cannot process any remaining operations, they do not affect subsequent scheduling and are irrelevant to the current scheduling. This information is not recorded in the state, which simplifies the state representation. Therefore, the state space gradually decreases as the scheduling proceeds (until it becomes empty, indicating that all operations have been scheduled), making this representation more efficient in real-time and large-scale applications.

[0062] Action: The set A(t) of all compatible action-machine pairs sets the action space for step t, specifically including: action a t ∈A(t) is set to the OM pair (O ij ,M k ), where O ij is qualified, that is, its direct potential operation has been completed, and M k ∈M ij for the idle;

[0063] State transfer: based on s t and d t , the environment updates the current operation and machine state to obtain the new state s t+1 ;

[0064] Reward: Reward r t The role of is to guide the agent to choose the operation that helps to minimize the maximum completion time of all operations C max action, at step t, estimate each operation O ij The completion time of the next term is C(O ij ,s t ), and C(O i0 )=0; if O ij Already dispatched, O ij The completion time is C ij ; For all unscheduled operations, the completion time of the next recursive approximation is:

[0065]

[0066] in, For operation O ij Processing time on machine k;

[0067] The reward r is the estimated total completion time difference between steps t and t+1. t The calculation formula is:

[0068]

[0069] In the scheduling process, the scheduling state is first converted into a heterogeneous graph structure; Figure 2 shown.

[0070] The heterogeneous graph structure includes heterogeneous disjunctive graph and competitive dynamic graph. The heterogeneous disjunctive graph structure is H = (O, M, C, ε), where the node set is the set of all operations, O S and O T To handle the start and end virtual nodes, the set C contains directed arcs representing the precedence constraints between consecutive operations of the same job, and the set M is the machine nodes, each of which corresponds to machine M. k , the set ε is the set of operation-machine arcs, where each element E ijk ∈ε is an undirected arc;

[0071] The scheduling process needs to pay attention not only to the global operation and machine status, but also to the local competition characteristics of the machines for the candidate operations at the current moment. It is a crucial challenge to prioritize the allocation of machines to candidate operations by mining the competition relationship between machines, because these machines may compete for multiple operation requests due to different processing capabilities, workloads, and availability.

[0072] Therefore, a competition dynamic graph structure D=(O,M,E) is proposed, where O is the set of current candidate operations, M is the set of candidate machines, and E is the set of arcs between operations and machines. The design of a dynamic competition graph can effectively capture the interaction between operations and machines and the diversity of machines. In the competition dynamic graph, nodes represent operations and machines respectively, and edges represent the connection and competition between them. For example, operation O 12 can be processed by machines m1 and m3, resulting in competition between the two machines. The structure of the dynamic competition graph builds and analyzes the graph structure to represent the relationship between operations and machines. The following attention mechanism aggregates information from neighbor nodes to accurately capture the complex interactions between the operations to be scheduled and the machines. This helps to optimize the allocation of operations in a multi-machine environment and improve the efficiency and overall performance of the manufacturing system.

[0073] This dual-isomorphic graph has two notable features. First, due to the dynamic changes in the graph topology caused by the Markov decision process (MDP) during scheduling, this topological difference provides a lot of useful information at the structural level. Second, most operation nodes in the graph have two different types of neighbors: the predecessor nodes of the operation and the predecessor nodes of the machine. The former reflects the priority constraints of the task, while the latter reflects the competing machines that can schedule the operation.

[0074] A two-stage embedding is performed using a heterogeneous graph attention network to extract the feature embedding of jobs and machines from heterogeneous graphs, including: taking node features as input and projecting the features of different types of nodes into the same feature space through a transformation matrix; using an improved dynamic graph attention mechanism to learn the attention weights between nodes and neighbors; and effectively adapting to the dynamic changes of the FJSP graph structure. This method enables the learned node embedding to better capture the complex structure and rich semantic information in the FJSP dual graph.

[0075] Each node is connected by different meta-path neighbors in the heterogeneous graph, and it is necessary to distinguish the subtle differences between them and select informative relationships. For each node, node-level attention aims to learn the importance of its meta-path-based neighbors and assign different importance values ​​to them. Node-level attention is set to mine the importance of each node's meta-path neighbors in the heterogeneous graph and aggregate the representations of these meaningful neighbors to form a node embedding. In this way, each node shows different importance according to its neighbors who play different roles in a specific task. Then, the node learns its embedding for each node before aggregating the information from its meta-path neighbors. As Figure 3 shown.

[0076] Taking node features as input and projecting the features of different types of nodes into the same feature space through the transformation matrix specifically includes the following steps:

[0077] For each type of node, the transformation matrix Project the features of different types of nodes into the same feature space, transform matrix Based on the node type, different feature spaces of various nodes are processed, and the projection process is:

[0078]

[0079] Among them, h i and h′ i are the original features and projection features of node i respectively, and generate a new set of node features:

[0080]

[0081] in, is the feature space after projecting h, F′ h is the projection operation on h.

[0082] Since GAT adopts a static attention mechanism, which limits the fitting ability of GAT on FJSP training data, GAT is introduced by modifying the operation order of GAT.

[0083] The improved dynamic graph attention mechanism is used to learn the attention weights between nodes and neighbors. Specifically,

[0084] Instantiation is handled by computing the weighted average of node representations. DGAT is proposed by modifying the internal operation order in GAT and integrating GATv2 to transform node features and handle all neighbors of node j in heterogeneous graphs.

[0085] Applying an A layer after the non-linearity and a W layer after the concatenation effectively applies a multi-layer perceptron to compute the score for each query-key pair:

[0086] e(h i ,h j )=a T LeakyReLU(W·[h i ||h j ]),

[0087] Among them, LeakyPeLU is nonlinear.

[0088] The dynamic graph attention model can associate elements in the input sequence and use the relationship between them to find important elements, making it very suitable for identifying high-priority operations and machines. In addition, the dynamic graph attention model can handle sequences of variable length (as input) and extract features of operations and machines, making it very suitable for solving FJSP instances of various sizes. Figure 4 shown.

[0089] The decision network uses feature embedding to generate action probability distribution, from which scheduling actions are sampled;

[0090] The decision network consists of the following:

[0091] Use DRL to train the agent by combining actor-critic and gradient policy with value function approximation;

[0092] Using the state features extracted from GAT, two multilayer perceptrons are set up as actors and critics with different parameters;

[0093] The actor network generates a probability distribution of actions based on the input features and determines the action selection of the agent in the current state; the critic network estimates the value function of the current state, indicating the expected cumulative reward of this state;

[0094] During training, the parameters θ of the actor network are updated through the policy gradient method; the parameters φ of the critic network are updated by minimizing the estimated error of the value function. This increases the probability of selecting high-value actions. At the same time, the parameters φ of the critic network are updated by minimizing the estimated error of the value function to make the value estimation more accurate. These networks gradually work together to optimize the decision-making ability of the agent in complex environments.

[0095] Use the proximal policy optimization algorithm to train the deep reinforcement learning model. Specifically include:

[0096] By using PPO-clip as a scheduling agent for training, the algorithm description of FJSP PPO-clip is shown in Algorithm 1 below.

[0097]

[0098]

[0099] For each training set, the environment and state are reset at the beginning; the simulation runs until a decision point t appears, and the current state of the system and the object queue are input into the PPO actor network; based on the current policy π, the SoftMax probability distribution of actions is output; the agent randomly samples actions according to this distribution and returns them to the environment;

[0100] A scheduling action is selected and which job in the queue should be processed; the simulation continues until the next decision point t+1 and the information of all jobs is updated again; the reward time of the action is calculated based on the idle time and fed back to the PPO;

[0101] At each step, record the action probability p(a) obtained by the critic network t ) and the state value function V(s t ), the trajectory (st ,a t ,r t ,V(s t ),p(a t )) is stored in the storage module;

[0102] After a fixed number of steps, the batch advantage function a is calculated using the generalized advantage estimate; the trajectory is divided into small batches, and the network parameters of the actor and critic are repeatedly updated using the data, and the new trajectory is used in the subsequent learning phase; the PPO scheduling agent uses the new strategy π new Interact with the environment.

[0103] PPO optimizes the decision network by limiting the probability ratio between the old and new policies, ensuring that the updates are not too drastic. This balance prevents large policy updates that could destabilize training. The parameters θ of the actor network are updated to maximize the pruned objective function, ensuring that the policy changes are within a safe range. At the same time, the parameters φ of the critic network are updated to minimize the value function estimation error. This dual optimization improves the estimation of both the policy and the value function, improving the performance of the agent. By iteratively optimizing the policy using PPO, the agent becomes more skilled at choosing actions that maximize the cumulative reward, while maintaining stability and efficiency during training.

[0104] The above-mentioned embodiments only express the specific implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the protection scope of the present application. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the technical solution concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. An effective reinforcement learning and graph neural network fusion algorithm for solving flexible shop scheduling, characterized by: The following steps are involved: The FJSP is formalized as an MDP by setting states, actions, state transitions, and rewards, while the scheduling process is described as dynamically allocating ready operations to compatible and idle machines; In the scheduling process, the scheduling state is first converted into a heterogeneous graph structure; A two-stage embedding is performed using a heterogeneous graph attention network to extract job and machine feature embedding from heterogeneous graphs, including: taking node features as input and projecting features of different types of nodes into the same feature space through a transformation matrix; using an improved dynamic graph attention mechanism to learn the attention weights between nodes and neighbors; The decision network uses feature embedding to generate action probability distribution, from which scheduling actions are sampled; Use the proximal policy optimization algorithm to train deep reinforcement learning models.

2. According to claim 1, an effective reinforcement learning and graph neural fusion algorithm for solving flexible shop scheduling is characterized in that: The node features are taken as input, and the features of different types of nodes are projected into the same feature space through the transformation matrix. The following steps are involved: For each type of node, the transformation matrix Project the features of different types of nodes into the same feature space, transform matrix Based on the node type, different feature spaces of various nodes are processed, and the projection process is: Among them, h i and h′ i are the original features and projection features of node i respectively, and generate a new set of node features: in, is the feature space after projecting h, F′ h is the projection operation on h.

3. According to claim 1, an effective reinforcement learning and graph neural fusion algorithm for solving flexible shop scheduling is characterized in that: The method of using the improved dynamic graph attention mechanism to learn the attention weights between nodes and neighbors specifically includes: Instantiation is handled by computing the weighted average of node representations. DGAT is proposed by modifying the internal operation order in GAT and integrating GATv2 to transform node features and handle all neighbors of node j in heterogeneous graphs. Applying an A layer after the non-linearity and a W layer after the concatenation effectively applies a multi-layer perceptron to compute the score for each query-key pair: e(h i h j )=a T LeakyReLU(W·[h i ||h j ]), Among them, LeakyReLU is nonlinear.

4. According to claim 1, an effective reinforcement learning and graph neural fusion algorithm for solving flexible shop scheduling is characterized in that: The decision network includes the following: Use DRL to train the agent by combining actor-critic and gradient policy with value function approximation; Using the state features extracted from GAT, two multilayer perceptrons are set up as actors and critics with different parameters; The actor network generates a probability distribution of actions based on the input features and determines the action selection of the agent in the current state; the critic network estimates the value function of the current state, indicating the expected cumulative reward of this state; During training, the parameters θ of the actor network are updated via the policy gradient method; the parameters φ of the critic network are updated by minimizing the estimated error of the value function.

5. According to claim 4, an effective reinforcement learning and graph neural fusion algorithm for solving flexible shop scheduling is characterized in that: The training of the deep reinforcement learning model using the proximal policy optimization algorithm specifically includes: Trained by using PPO-clip as a scheduling agent; For each training set, the environment and state are reset at the beginning; the simulation runs until a decision point t appears, and the current state of the system and the object queue are input into the PPO actor network; based on the current policy π, the SoftMax probability distribution of actions is output; the agent randomly samples actions according to this distribution and returns them to the environment; A scheduling action is selected and which job in the queue should be processed; the simulation continues until the next decision point t+1 and the information of all jobs is updated again; the reward time of the action is calculated based on the idle time and fed back to the PPO; At each step, record the action probability p(a) obtained by the critic network t ) and the state value function V(s t ), the trajectory (s t ,a t ,r t ,V(s t ),p(a t )) is stored in the storage module; After a fixed number of steps, the batch advantage function a is calculated using the generalized advantage estimate; the trajectory is divided into small batches, and the network parameters of the actor and critic are repeatedly updated using the data, and the new trajectory is used in the subsequent learning phase; the PPO scheduling agent uses the new strategy π new Interact with the environment.

6. According to claim 1, an effective reinforcement learning and graph neural fusion algorithm for solving flexible shop scheduling is characterized in that: The FJSP is formalized into an MDP by setting states, actions, state transitions and rewards. In step t, the DRL model receives a state s from the environment. t and take action a t , assigns compatible actions to machines for immediate processing, with processing time Ts(t), and the FJSP environment returns a reward r associated with the completion time t ,Repeat the process for all operations |O| with heavy tasks and obtain the solution of FJSP.

7. According to claim 6, an effective reinforcement learning and graph neural fusion algorithm for solving flexible shop scheduling is characterized in that: The MDP settings include: Status: Status t ∈S contains all operations and machine features related to the current decision of step t; where s0 is the initial state and each state is described by a heterogeneous graph; Action: The set A(t) of all compatible action-machine pairs sets the action space for step t, specifically including: action a t ∈A(t) is set to the OM pair (O ij ,M k ), where O ij is qualified, that is, its direct potential operation has been completed, and M k ∈M ij for the idle; State transfer: based on s t and a t , the environment updates the current operation and machine state to obtain the new state s t+1 ; Reward: Reward r t The role of is to guide the agent to choose the operation that helps to minimize the maximum completion time of all operations C max action, at step t, estimate each operation O ij The completion time of the next term is C(O ij ,s t ), and C(O i0 )=0; if O ij Already dispatched, O ij The completion time is C ij ; For all unscheduled operations, the completion time of the next recursive approximation is: in, For operation O ij Processing time on machine k; The reward r is the estimated total completion time difference between steps t and t+1. t The calculation formula is:

8. According to claim 1, an effective reinforcement learning and graph neural fusion algorithm for solving flexible shop scheduling is characterized in that: The heterogeneous graph structure includes a heterogeneous disjunctive graph and a competitive dynamic graph. The heterogeneous disjunctive graph structure is H=(O, M, C, ε), where the node set is the set of all operations, O S and O T To handle the start and end virtual nodes, the set C contains directed arcs representing the precedence constraints between consecutive operations of the same job, and the set M is the machine nodes, each of which corresponds to machine M. k , the set ε is the set of operation-machine arcs, where each element E ijk ∈ε is an undirected arc; The structure of the competition dynamic graph is D=(O,M,E), where O is the set of current candidate operations, M is the set of candidate machines, and E is the set of arcs between operations and machines. In the competition dynamic graph, nodes represent operations and machines respectively, and edges represent the connections and competitions between them.

Images