An effective reinforcement learning and graph neural network fusion method for solving flexible shop floor scheduling

Abstract

This invention discloses an effective reinforcement learning and graph neural network fusion method for solving flexible job scheduling, comprising the following steps: formalizing the Flexible Job Scheduler (FJSP) into a Multi-Level Product (MDP) by setting states, actions, state transitions, and rewards, while describing the scheduling process as dynamically allocating ready operations to compatible and idle machines; during the scheduling process, the scheduling state is first transformed into a heterogeneous graph structure; a heterogeneous graph attention network is used for two-stage embedding to extract feature embeddings of jobs and machines from the heterogeneous graph; a decision network uses the feature embeddings to generate an action probability distribution, from which scheduling actions are sampled; and a deep reinforcement learning model is trained using a proximal policy optimization algorithm. Compared with traditional PDR and SOTA DRL methods, the algorithm achieves significantly high-quality solutions on two synthetic datasets with different distributions, improving performance and generalization ability. It also exhibits good generalization ability because the model is trained on a small number of instances on a large number of or out-of-distribution instances.

Description

Technical Field

[0001] This invention relates to the field of flexible shop scheduling technology, specifically to an effective reinforcement learning and graph neural network fusion method for solving flexible shop scheduling problems. Background Technology

[0002] The Flexible Job Shop Scheduling (FJSP) problem is widely used in various fields such as semiconductor manufacturing, automobile production, and textiles to maintain production efficiency and cost-effectiveness. With the rapid development of information technology, FJSP has gradually become a hot topic for flexible manufacturing companies because it is more flexible and complex than the Relaxed Job Shop Scheduling (JSSP) problem, which allows each operation to be processed on multiple compatible machines.

[0003] In recent years, Deep Reinforcement Learning (DRL) has become a new research hotspot in Job Shop Scheduling (JSP) due to its ability to rapidly respond to dynamic environments and make online decisions. Reinforcement learning-based scheduling agents can interact with their environment and automatically learn optimal policies, making them suitable for solving stochastic, dynamic, multi-stage sequential decision-making problems in JSP. Typically, parameterized neural network models take information from the production environment as input and output a probability distribution of feasible scheduling actions, forming an end-to-end learning method. By training on production process data and modeling the scheduling process as a Markov Decision Process (MDP), the DRL model can adaptively select the optimal action based on the current state to maximize the total reward associated with the FJSP production objective. The disjunctive graph introduced for Job Shop Scheduling (JSP) effectively represents instances of the Flexible Job Shop Scheduling (FJSP) problem, representing the sequential constraints between jobs and the situation where multiple operations are processed on the same machine. Recent research has shown that graph neural networks (GNNs) can effectively find solutions under complex constraints after modeling the scheduling problem as a disjunctive graph. Although the relationships between operation nodes are modeled through the disjunctive graph, the relationships between machines and operations have not yet been modeled in the graph to fully utilize the complexities of FJSP. Currently, heterogeneous graphs are introduced to model the relationship between operations and machines by attaching a set of machine nodes to the original disjoint graph. To further model the competition among machines for unscheduled operations, graph attention networks (GAT) are introduced to learn the embeddings of machines and operations separately.

[0004] Facing complex FJSPs of various sizes, there are still three main challenges in effectively solving them using deep reinforcement learning (DRL) methods: (1) how to reasonably model the competitive relationship between compatible machines; (2) how to extract global and local scheduling information through deep neural networks to obtain a more accurate state representation; and (3) how to capture the dynamic evolution scheduling characteristics to effectively solve FJSPs of different sizes. Summary of the Invention

[0005] The purpose of this invention is to provide an effective reinforcement learning and graph neural network fusion method for solving flexible workshop scheduling, addressing the aforementioned problems. This method models operations and machines from both global and local perspectives using a dual heterogeneous graph, and learns state features through an attention mechanism to better capture the dynamic features of graphs with strong expressive power.

[0006] The technical solution of the present invention is as follows:

[0007] An effective reinforcement learning and graph neural network fusion method for solving flexible shop floor scheduling includes the following steps:

[0008] The FJSP is formalized into an MDP by setting states, actions, state transitions and rewards, while the scheduling process is described as dynamically allocating ready operations to compatible and idle machines.

[0009] During the scheduling process, the scheduling state is first transformed into a heterogeneous graph structure;

[0010] A two-stage embedding is performed using a heterogeneous graph attention network to extract feature embeddings of jobs and machines from the heterogeneous graph. This includes: taking node features as input and projecting the features of different types of nodes to the same feature space through a transformation matrix; and learning the attention weights between nodes and their neighbors using an improved dynamic graph attention mechanism.

[0011] The decision network uses feature embeddings to generate action probability distributions, from which actions are sampled and scheduled.

[0012] Training deep reinforcement learning models using a proximal policy optimization algorithm.

[0013] Furthermore, the step of taking node features as input and projecting the features of different types of nodes onto the same feature space through a transformation matrix specifically includes the following steps:

[0014] For each type of node, the transformation matrix is ​​used. Projecting the features of different types of nodes onto the same feature space, the transformation matrix... Based on the node type, different feature spaces of various nodes are processed, and the projection process is as follows:

[0015] ,

[0016] in, and They are nodes The original features and projected features are used to generate a new set of node features:

[0017] ,

[0018] in, To The feature space after projection. To Perform a projection operation.

[0019] Furthermore, the method of learning attention weights between nodes and their neighbors using the improved dynamic graph attention mechanism specifically includes:

[0020] Instantiation is handled by calculating a weighted average of node representations; DGAT is proposed by modifying the internal operation order in GAT and integrating GATv2 to transform node features and handle nodes in heterogeneous graphs. All neighbors;

[0021] Applications after nonlinearity Layers, and apply them after connection. Layers effectively apply a multilayer perceptron to compute a score for each query-key pair:

[0022] ,

[0023] in, It is nonlinear.

[0024] Furthermore, the decision network includes the following:

[0025] The agent is trained using DRL, and approximation of the value function is achieved by combining actor-critic and gradient policies;

[0026] Using state features extracted from GAT, two multilayer perceptrons are set up as actors and critics, with different parameters;

[0027] The actor network generates a probability distribution of actions based on input features and determines the agent's action choice in the current state; the critic network estimates the value function of the current state, indicating the expected cumulative reward of that state.

[0028] During training, the parameters of the participant network Updated using the policy gradient method; parameters of the critic network. Update the value function by minimizing the estimation error.

[0029] Furthermore, the training of the deep reinforcement learning model using the proximal policy optimization algorithm specifically includes:

[0030] Training was conducted using PPO-clip as a scheduling agent;

[0031] For each training set, the environment and state are reset at the start; the simulation runs until the decision point. The system's current state and object queue are input into the PPO actor network; based on the current policy... The output action is a SoftMax probability distribution; the agent randomly samples actions based on this distribution and returns them to the environment.

[0032] Select a scheduling operation and determine which job in the queue should be processed; the simulation continues until the next decision point. And update all job information again; calculate the reward time for actions based on idle time and report it to PPO;

[0033] Record the action probabilities obtained by the commentator network at each step. and state value function , trajectory Stored in the storage module;

[0034] After a fixed number of steps, the batch dominance function is calculated using generalized dominance estimation. The trajectory is divided into small batches, and the network parameters of the actors and critics are repeatedly updated using the data. The new trajectories are used in subsequent learning phases. The PPO scheduling agent uses a new strategy. Interact with the environment.

[0035] Furthermore, the formalization of FJSP into MDP by setting states, actions, state transitions, and rewards includes: in the following steps The DRL model receives state from the environment. And take action To ensure immediate processing and allocation of compatible functions to the machine, the processing time is... The FJSP environment returns a reward related to the completion time. All operations on heavy tasks Repeat the process to obtain the FJSP solution.

[0036] Furthermore, the MDP settings include:

[0037] State: State Includes and steps All operations and machine characteristics relevant to the current decision; among which, The initial state is defined by a heterogeneous graph;

[0038] Action: A set of all compatible operation-machine pairs Steps were set. The action space specifically includes: actions Set in the step Feasible right ,in To be considered qualified, meaning that its direct potential operation has been completed, and Available;

[0039] State transition: based on and The environment updates the current operation and machine state to obtain the new state. ;

[0040] Rewards: Rewards Its function is to guide the agent in making choices that help minimize the maximum completion time of all operations. The action, in the steps Estimate each operation The completion time for the next session is ,and ;if It has been scheduled. The completion time is For all scheduled operations, their completion time can be recursively approximated as follows:

[0041] ,

[0042] in, For operation In the machine Processing time;

[0043] As a step and The estimated total completion time difference between the two, and the reward The calculation formula is:

[0044] .

[0045] Furthermore, the heterogeneous graph structure includes a heterogeneous disjunction graph and a competing dynamic graph, wherein the heterogeneous disjunction graph structure is as follows: , where the set of nodes For the set of all operations, and To handle the start and end virtual nodes, the set A set containing directed arcs representing priority constraints between consecutive operations of the same job. For machine nodes, each machine node corresponds to a machine. ,gather For the set of operation-machine arcs, where each element It is an undirected arc;

[0046] The competitive dynamic graph structure is as follows ,in The set of current candidate operations. For the set of candidate machines, Given the set of arcs between operations and machines, in the competition dynamic graph, nodes represent operations and machines respectively, and edges represent the connections and competition between them.

[0047] Compared with existing technologies, the advantages of this invention are:

[0048] 1. An effective reinforcement learning and graph neural network fusion method for solving flexible shop floor scheduling is proposed. An innovative end-to-end DRL algorithm is proposed, which models operations and machines from both global and local perspectives through dual heterogeneous graphs. Specifically, the heterogeneous disjunctive graph describes the entire operation and machine nodes to capture global features, while the dynamic competition graph encodes the current candidate nodes and possible machines to capture local features.

[0049] 2. An effective reinforcement learning and graph neural network fusion method for solving flexible workshop scheduling. Unlike the traditional GAT, a new attention mechanism is proposed to learn state features, which can better capture the dynamic features of graphs with strong expressive power.

[0050] 3. An effective reinforcement learning and graph neural network fusion method for solving flexible shop floor scheduling. Compared with traditional PDR and SOTA DRL methods, the algorithm in this application achieves significantly high-quality solutions on two synthetic datasets with different distributions, improving performance and generalization ability. Furthermore, it exhibits good generalization ability because the model is trained on a small number of instances on a large number of instances or out-of-distribution instances. Attached Figure Description

[0051] Figure 1 This is a framework diagram of the algorithm in this application.

[0052] Figure 2 This is the heterogeneous disjunction graph of the algorithm in this application.

[0053] Figure 3 This is the dynamic competition graph for the algorithm in this application.

[0054] Figure 4 This is a network architecture diagram of the algorithm in this application. Detailed Implementation

[0055] It should be noted that relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0056] The features and performance of the present invention will be further described in detail below with reference to embodiments.

[0057] Please see Figure 1-4 An effective reinforcement learning and graph neural network fusion method for solving flexible shop floor scheduling, such as Figure 1 Therefore, it includes the following steps:

[0058] The FJSP is formalized into a Markov Decision Process (MDP) by setting states, actions, state transitions, and rewards, while the scheduling process is described as dynamically allocating ready operations to compatible and idle machines.

[0059] Formalizing FJSP into MDP by setting states, actions, state transitions, and rewards includes the following steps: The DRL model receives state from the environment. And take action To ensure immediate processing and allocation of compatible functions to the machine, the processing time is... The FJSP environment returns a reward related to the completion time. All operations on heavy tasks Repeat the process to obtain the FJSP solution.

[0060] MDP settings include:

[0061] State: State Includes and steps All operations and machine characteristics relevant to the current decision; among which, The initial state is represented by a heterogeneous graph; since completed operations and the machine cannot process any remaining operations, they do not affect subsequent scheduling and are irrelevant to the current scheduling. This information is not recorded in the state, thus simplifying the state representation. Therefore, the state space gradually decreases as scheduling progresses (until it becomes empty, indicating that all operations have been scheduled), making this representation more efficient in real-time and large-scale applications.

[0062] Action: A set of all compatible operation-machine pairs Steps were set. The action space specifically includes: actions Set in the step Feasible right ,in To be considered qualified, meaning that its direct potential operation has been completed, and Available;

[0063] State transition: based on and The environment updates the current operation and machine state to obtain the new state. ;

[0064] Rewards: Rewards Its function is to guide the agent in making choices that help minimize the maximum completion time of all operations. The action, in the steps Estimate each operation The completion time for the next session is ,and ;if It has been scheduled. The completion time is For all scheduled operations, their completion time can be recursively approximated as follows:

[0065] ,

[0066] in, For operation In the machine Processing time;

[0067] As a step and The estimated total completion time difference between the two, and the reward The calculation formula is:

[0068] .

[0069] During the scheduling process, the scheduling state is first transformed into a heterogeneous graph structure; such as Figure 2 As shown.

[0070] Heterogeneous graph structures include heterogeneous disjunction graphs and competing dynamic graphs. The heterogeneous disjunction graph structure is as follows: , where the set of nodes For the set of all operations, and To handle the start and end virtual nodes, the set A set containing directed arcs representing priority constraints between consecutive operations of the same job. For machine nodes, each machine node corresponds to a machine. ,gather For the set of operation-machine arcs, where each element It is an undirected arc;

[0071] The scheduling process needs to consider not only the global operations and machine states, but also the local competition characteristics of the machines competing for candidate operations at the current moment. Mining the competition relationships between machines to prioritize the allocation of candidate operations is a crucial challenge, as these machines may compete for multiple operation requests due to varying processing power, workload, and availability.

[0072] Therefore, a competitive dynamic graph structure is proposed as follows: ,in The set of current candidate operations. For the set of candidate machines, For a set of arcs between operations and machines, a dynamic competition graph can be designed to effectively capture the interactions between operations and machines, as well as the diversity of machines. In the competition dynamic graph, nodes represent operations and machines, and edges represent the connections and competition between them. For example, operations... Can be made by machine and This process generates competition between the two machines. A dynamic competition graph is constructed and analyzed to represent the relationships between operations and machines. Information is aggregated from neighboring nodes using an attention mechanism to accurately capture the complex interactions between the scheduled operations and the machines. This helps optimize operation allocation in multi-machine environments, improving the efficiency and overall performance of manufacturing systems.

[0073] This biisomorphic graph has two significant features. First, the dynamic changes in graph topology caused by Markov Decision Processes (MDPs) during scheduling provide a wealth of useful information at the structural level. Second, most operation nodes in the graph have two different types of neighbors: the operation's predecessor node and the machine's predecessor node. The former reflects the priority constraints of the task, while the latter reflects the competing machines that can schedule the operation.

[0074] A two-stage embedding method using a heterogeneous graph attention network is employed to extract feature embeddings for jobs and machines from the heterogeneous graph. This includes: taking node features as input and projecting features of different types of nodes onto the same feature space using a transformation matrix; learning attention weights between nodes and their neighbors using an improved dynamic graph attention mechanism; and effectively adapting to the dynamic changes in the FJSP graph structure. This method enables the learned node embeddings to better capture the complex structure and rich semantic information in the FJSP bi-graph.

[0075] Each node is connected through different meta-path neighbors in the heterogeneous graph, necessitating the differentiation of subtle differences and the selection of informative relationships. For each node, node-level attention aims to learn the importance of its meta-path-based neighbors and assign them distinct importance values. Node-level attention is configured to mine the importance of each node's meta-path neighbors in the heterogeneous graph and aggregate the representations of these meaningful neighbors to form node embeddings. In this way, each node exhibits different importance based on its neighbors, which play different roles in a specific task. Then, each node learns its embedding for each other before aggregating information from its meta-path neighbors. Figure 3 As shown.

[0076] Using node features as input and projecting the features of different types of nodes onto the same feature space through a transformation matrix specifically includes the following steps:

[0077] For each type of node, the transformation matrix is ​​used. Projecting the features of different types of nodes onto the same feature space, the transformation matrix... Based on the node type, different feature spaces of various nodes are processed, and the projection process is as follows:

[0078] ,

[0079] in, and They are nodes The original features and projected features are used to generate a new set of node features:

[0080] ,

[0081] in, To The feature space after projection. To Perform a projection operation.

[0082] Since GAT uses a static attention mechanism, this limits its ability to fit FJSP training data. Therefore, GAT was introduced by modifying the order of operations in GAT.

[0083] The improved dynamic graph attention mechanism learns the attention weights between nodes and their neighbors, specifically including:

[0084] Instantiation is handled by calculating a weighted average of node representations; DGAT is proposed by modifying the internal operation order in GAT and integrating GATv2 to transform node features and handle nodes in heterogeneous graphs. All neighbors;

[0085] Applications after nonlinearity Layers, and apply them after connection. Layers effectively apply a multilayer perceptron to compute a score for each query-key pair:

[0086] ,

[0087] in, It is nonlinear.

[0088] Dynamic graph attention models can associate elements in an input sequence and leverage the relationships between them to find important elements, making them well-suited for identifying high-priority operations and machines. Furthermore, dynamic graph attention models can handle sequences of variable length (as input), extracting features of operations and machines, thus making them ideal for solving FJSP instances of various sizes. Figure 4 As shown.

[0089] The decision network uses feature embeddings to generate action probability distributions, from which actions are sampled and scheduled.

[0090] Decision networks include the following:

[0091] The agent is trained using DRL, and approximation of the value function is achieved by combining actor-critic and gradient policies;

[0092] Using state features extracted from GAT, two multilayer perceptrons are set up as actors and critics, with different parameters;

[0093] The actor network generates a probability distribution of actions based on input features and determines the agent's action choice in the current state; the critic network estimates the value function of the current state, indicating the expected cumulative reward of that state.

[0094] During training, the parameters of the participant network Updated using the policy gradient method; parameters of the critic network. Updates are made by minimizing the estimation error of the value function to increase the probability of selecting high-value actions. Simultaneously, the parameters of the commentator network... The value estimate is updated by minimizing the estimation error of the value function, thus making the value estimate more accurate. These networks gradually work together to optimize the agent's decision-making ability in complex environments.

[0095] Training deep reinforcement learning models using a proximal policy optimization algorithm. Specifically, this includes:

[0096] The algorithm description of FJSP PPO-clip is as follows: Algorithm 1, which uses PPO-clip as a scheduling agent for training.

[0097]

[0098] For each training set, the environment and state are reset at the start; the simulation runs until the decision point. The system's current state and object queue are input into the PPO actor network; based on the current policy... The output action is a SoftMax probability distribution; the agent randomly samples actions based on this distribution and returns them to the environment.

[0099] Select a scheduling operation and determine which job in the queue should be processed; the simulation continues until the next decision point. And update all job information again; calculate the reward time for actions based on idle time and report it to PPO;

[0100] Record the action probabilities obtained by the commentator network at each step. and state value function , trajectory Stored in the storage module;

[0101] After a fixed number of steps, the batch dominance function is calculated using generalized dominance estimation. The trajectory is divided into small batches, and the network parameters of the actors and critics are repeatedly updated using the data. The new trajectories are used in subsequent learning phases. The PPO scheduling agent uses a new strategy. Interact with the environment.

[0102] PPO optimizes the decision network by limiting the probability ratio between new and old policies, ensuring that updates are not too drastic. This balance prevents large-scale policy updates that could lead to unstable training. (Parameters of the actor network) The parameters are updated to maximize the pruned objective function, ensuring policy changes remain within a safe range. Meanwhile, the critic network's parameters... The policy is updated to minimize the value function estimation error. This dual optimization improves both the policy and value function estimation, thus enhancing the agent's performance. By iteratively optimizing the policy using PPO, the agent becomes more adept at selecting actions that maximize cumulative rewards, while maintaining stability and efficiency during training.

[0103] The embodiments described above merely illustrate specific implementation methods of this application, and while the descriptions are detailed and specific, they should not be construed as limiting the scope of protection of this application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the technical solution of this application, and these modifications and improvements all fall within the scope of protection of this application.

Claims

1. An effective reinforcement learning and graph neural network fusion method for solving flexible shop floor scheduling, characterized in that, Includes the following steps: The FJSP is formalized into an MDP by setting states, actions, state transitions and rewards, while the scheduling process is described as dynamically allocating ready operations to compatible and idle machines. During the scheduling process, the scheduling state is first transformed into a heterogeneous graph structure; A two-stage embedding is performed using a heterogeneous graph attention network to extract feature embeddings of jobs and machines from the heterogeneous graph. This includes: taking node features as input and projecting the features of different types of nodes to the same feature space through a transformation matrix; and learning the attention weights between nodes and their neighbors using an improved dynamic graph attention mechanism. The decision network uses feature embeddings to generate action probability distributions, from which actions are sampled and scheduled. Training deep reinforcement learning models using a proximal policy optimization algorithm; Heterogeneous graph structures include heterogeneous disjunction graphs and competition dynamic graphs. Disjunction graphs represent sequence constraints between jobs and the processing of multiple operations on the same machine. The heterogeneous disjunction graph structure is... , where the set of nodes For the set of all operations, and To handle the start and end virtual nodes, the set A set containing directed arcs representing priority constraints between consecutive operations of the same job. For machine nodes, each machine node corresponds to a machine. ,gather For the set of operation-machine arcs, where each element It is an undirected arc; The competitive dynamic graph structure is as follows ,in The set of current candidate operations. For the set of candidate machines, In the competition dynamic graph, nodes represent operations and machines, and edges represent connections and competition between them.

2. The effective reinforcement learning and graph neural network fusion method for solving flexible shop floor scheduling according to claim 1, characterized in that, The process of taking node features as input and projecting the features of different types of nodes onto the same feature space using a transformation matrix is ​​specifically... Includes the following steps: For each type of node, the transformation matrix is ​​used. Projecting the features of different types of nodes onto the same feature space, the transformation matrix... Based on the node type, different feature spaces of various nodes are processed, and the projection process is as follows: ， in, and They are nodes The original features and projected features are used to generate a new set of node features: ， in, To The feature space after projection. To Perform a projection operation. The number of nodes.

3. The effective reinforcement learning and graph neural network fusion method for solving flexible shop floor scheduling according to claim 1, characterized in that, The improved dynamic graph attention mechanism for learning attention weights between nodes and their neighbors specifically includes: Instantiation is handled by calculating a weighted average of node representations; DGAT is proposed by modifying the internal operation order in GAT and integrating GATv2 to transform node features and handle nodes in heterogeneous graphs. All neighbors; Applications after nonlinearity Layers, and apply them after connection. Layers effectively apply a multilayer perceptron to compute a score for each query-key pair: , in, It is nonlinear. For nodes The original characteristics, For nodes The original characteristics.

4. The effective reinforcement learning and graph neural network fusion method for solving flexible shop floor scheduling according to claim 1, characterized in that, The decision-making network includes the following: The agent is trained using DRL, and approximation of the value function is achieved by combining actor-critic and gradient policies; Using state features extracted from GAT, two multilayer perceptrons are set up as actors and critics, with different parameters; The actor network generates a probability distribution of actions based on input features and determines the agent's action choice in the current state; the critic network estimates the value function of the current state, indicating the expected cumulative reward of that state. During training, the parameters of the participant network Updated using the policy gradient method; parameters of the critic network. The update is performed by minimizing the estimation error of the value function.

5. The effective reinforcement learning and graph neural network fusion method for solving flexible shop floor scheduling according to claim 4, characterized in that, The training of the deep reinforcement learning model using the proximal policy optimization algorithm specifically includes: Training was conducted using PPO-clip as a scheduling agent; For each training set, the environment and state are reset at the start; the simulation runs until the decision point. The system's current state and object queue are input into the PPO actor network; based on the current policy... The output action is a SoftMax probability distribution; the agent randomly samples actions based on this distribution and returns them to the environment. Select a scheduling operation and determine which job in the queue should be processed; the simulation continues until the next decision point. And update all job information again; calculate the reward time for actions based on idle time and report it back to PPO; Record the action probabilities obtained by the commentator network at each step. and state value function , trajectory Stored in the storage module, where, Let t be the state at decision point t. For the action at decision point t, Let the decision point be at time t. implement The rewards received; After a fixed number of steps, the batch dominance function is calculated using generalized dominance estimation. The trajectory is divided into small batches, and the network parameters of the actors and critics are repeatedly updated using the data. The new trajectories are used in subsequent learning phases. The PPO scheduling agent uses a new strategy. Interact with the environment.

6. The effective reinforcement learning and graph neural network fusion method for solving flexible shop floor scheduling according to claim 1, characterized in that, The formalization of FJSP into MDP by setting states, actions, state transitions, and rewards includes: at the decision point The DRL model receives state from the environment. And take action To ensure immediate processing and allocation of compatible functions to the machine, the processing time is... The FJSP environment returns a reward related to the completion time. All operations on heavy tasks Repeat the process to obtain the FJSP solution.

7. The effective reinforcement learning and graph neural network fusion method for solving flexible shop floor scheduling according to claim 6, characterized in that, The MDP settings include: State: State Includes decision points All operations and machine characteristics relevant to the current decision; among which, The initial state is defined by a heterogeneous graph; Action: A set of all compatible operation-machine pairs Decision points were set. The action space specifically includes: actions Set at the decision point Feasible right ,in To be considered qualified, meaning that its direct potential operation has been completed, and For free, for The set of optional machines; State transition: based on and The environment updates the current operation and machine state to obtain the new state. ; Rewards: Rewards Its function is to guide the agent in making choices that help minimize the maximum completion time of all operations. Actions at the decision point Estimate each operation The completion time for the next session is ,and ;if It has been scheduled. The completion time is For all unscheduled operations, their completion time can be recursively approximated as follows: ， in, For operation In the machine Processing time; As a decision point and The estimated total completion time difference between the two, and the reward The calculation formula is: 。

Images