Job shop scheduling method based on heterogeneous graph neural network and mask reinforcement learning
By employing heterogeneous graph neural networks and masked reinforcement learning, this study addresses the challenges of representing the relationship between processes and machines and ensuring the legality of constraints in large-scale job shop scheduling problems. This approach achieves efficient and accurate scheduling optimization, significantly improving solution efficiency and solution quality.
Patent Information
- Application Number
- CN202610387794.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-03-27
- Publication Date
- 2026-06-26
AI Technical Summary
Existing technologies struggle to effectively characterize the complex topological relationships and dynamic constraints between processes and machines when dealing with large-scale job shop scheduling problems, and cannot guarantee the legality of the process rules in the scheduling scheme, resulting in low solution efficiency.
A heterogeneous graph neural network and masked reinforcement learning approach is adopted. By constructing a heterogeneous graph data object containing processes and machine nodes, action mask vectors are generated. The heterogeneous graph convolution mechanism and policy network are combined to output action probabilities. The model parameters are updated using a proximal policy optimization algorithm to ensure the legality of actions. Illegal actions are blocked through a Petri net-style dynamic masking mechanism.
It achieves efficient and accurate job shop scheduling, and can quickly generate optimized solutions that meet process constraints, significantly improving solution efficiency and scheduling quality.
Smart Images
Figure CN122288237A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of intelligent manufacturing and operations research optimization, and in particular to a job shop scheduling method based on heterogeneous graph neural networks and masked reinforcement learning. Background Technology
[0002] The Job Shop Scheduling Problem (JSSP) is a classic NP-hard combinatorial optimization problem in manufacturing production operations. Its core requirement is to rationally arrange the start time of each process under the dual conditions of satisfying process constraints (process sequence) and resource constraints (machine occupancy limit) to ultimately achieve the goal of minimizing the maximum completion time.
[0003] Traditional solution methods, such as branch and bound and genetic algorithms, can achieve good solutions for small-scale scheduling problems. However, when faced with large-scale, multi-constraint real-world production scenarios, they often suffer from bottlenecks such as excessive computation time or getting trapped in local optima. In recent years, deep reinforcement learning (DRL)-based methods have shown great application potential due to their end-to-end learning capabilities. However, existing DRL methods still face two major challenges when dealing with JSSP: first, it is difficult to effectively represent the complex topological relationships and dynamic constraints between processes and machines; second, it cannot guarantee that the actions output by the model always meet strict process rules, easily generating invalid scheduling schemes and resulting in low solution efficiency. Summary of the Invention
[0004] This invention aims to provide a job shop scheduling method based on heterogeneous graph neural networks and masked reinforcement learning, which solves the problems of low efficiency and poor robustness of traditional methods, as well as the difficulty of existing reinforcement learning methods in handling complex constraints and the tendency to generate invalid actions.
[0005] To achieve the above objectives, the technical solution of the present invention includes:
[0006] A job shop scheduling method based on heterogeneous graph neural networks and masked reinforcement learning includes the following steps:
[0007] S1: Read instance data of the job shop scheduling problem, parse job, process and machine information, and construct a structured instance object containing a list of process objects and a job index mapping table;
[0008] S2: In an event-driven simulation environment, the current scheduling state is extracted and modeled as a heterogeneous graph data object containing process nodes and machine nodes;
[0009] S3: In each decision step, traverse the unfinished processes and generate the action mask vector of the corresponding process set based on the completion status of the process's predecessor dependencies and the idle status of the specified processing machine.
[0010] S4: Input the heterogeneous graph data object into the heterogeneous graph neural network model, aggregate node features through the heterogeneous graph convolution mechanism, and output the action probability corresponding to each process and the value assessment scalar of the current state.
[0011] S5: Combine the action mask vector with the action probability to perform action sampling; use the near-end policy optimization algorithm to interact with the simulation environment to collect trajectory data, and update the network parameters of the heterogeneous graph neural network model with the goal of minimizing the maximum completion time to obtain the scheduling optimization model;
[0012] S6: Use the scheduling optimization model to reason about the new scheduling instance, and select the operation with the highest probability of action to execute at each step to generate the optimal operation.
[0013] A further improvement of the present invention is that the process of constructing the heterogeneous graph in step S2 includes:
[0014] Establish process sequence edges to constrain the order of processes;
[0015] Establish equipment association edges to constrain the association relationship between process nodes and machine nodes.
[0016] A further improvement of the present invention is that the process node features include normalized processing time, job number position code, machine number position code, and ready status flag; the machine node features include machine release time and idle status flag.
[0017] A further improvement of the present invention is that the feature acquisition process of the process node includes:
[0018] Extract the processing time of each process and use min-max normalization to process the input feature vector;
[0019] A location coding algorithm is used to generate location codes for the job number and machine number.
[0020] A further improvement of the present invention is that, in step S3, during the generation of the action mask vector:
[0021] Check the completion flag of the preceding process of each process to determine whether it meets the ready condition. If all the preceding processes of an incomplete process have been completed or the process is the first process of the operation and has not started processing, then set the corresponding position in the action mask vector to 1; otherwise, set it to 0.
[0022] If the action mask vector generated in the current decision step is all 0, it is determined that there is no legal action at the current moment. The simulation environment triggers a priority queue mechanism at the bottom layer, automatically advances the environment time to the moment of the next nearest process completion event, and re-executes state modeling and mask generation.
[0023] A further improvement of the present invention is that the heterogeneous graph neural network model includes:
[0024] Heterogeneous graph convolutional layer: Internally configured graph attention convolutional layers to enhance feature weights. Each convolutional output is connected to a non-linear activation layer and a normalization layer. After several convolutional layers, deep feature embeddings of process nodes and machine nodes are obtained respectively.
[0025] Decoding layer: Includes policy network and value network. The policy network outputs an action probability distribution with the same dimension as the total number of processes through a fully connected layer and a softmax function. The value network outputs a scalar value assessment of the current state through global average pooling by a fully connected layer.
[0026] A further improvement of the present invention is that, in step S5, before action sampling, the action probability distribution output by the policy network is combined with the action mask, and the action probability of illegal procedures with corresponding action mask values of 0 is set to negative infinity, so as to ensure that action sampling is performed only from legal procedures based on the action probability distribution.
[0027] The intelligent agent performs sampling actions to interact with the simulation environment and collects trajectory data including heterogeneous graph data, action masks, selected actions, and environmental rewards.
[0028] Based on the collected trajectory data, the generalized advantage estimation algorithm is used to calculate the advantage function, and the backpropagation mechanism of the proximal policy optimization algorithm is used to synchronously update the parameters of the policy network, value network and heterogeneous graph neural network in reinforcement learning.
[0029] A further improvement of the present invention is that step S6 further includes generating a JSON format scheduling file containing information such as process number, machine number, start and end time according to the optimal process; and drawing a Gantt chart using a visualization library.
[0030] Compared to existing technologies, this method exhibits superior structure awareness. Employing heterogeneous graph neural networks to model scheduling states, it naturally adapts to the heterogeneous attributes of processes and machines, accurately depicting the complex relationships between them. This approach better aligns with the essence of JSSP problems compared to traditional feature encoding methods. The method achieves a balance between efficiency and accuracy. Based on the PPO end-to-end learning framework, it implements a one-time training, fast inference application mode. After training, its speed in solving new scheduling instances far surpasses traditional heuristic algorithms, while also providing superior scheduling quality. Attached Figure Description
[0031] Figure 1 This is an overall flowchart of the method described in this invention;
[0032] Figure 2 This is a schematic diagram illustrating the construction of the heterogeneous graph state in this invention;
[0033] Figure 3 This is a model architecture diagram of a heterogeneous graph neural network (Hetero-GNN);
[0034] Figure 4 This is a diagram illustrating the interaction logic between Petri net style mask generation and PPO sampling.
[0035] Figure 5 Example of a Gantt chart for visualizing scheduling results. Detailed Implementation
[0036] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. These embodiments are based on the technical solution of the present invention and provide detailed implementation methods and specific operating procedures. However, the scope of protection of the present invention is not limited to the following embodiments.
[0037] like Figure 1 , Figure 2 As shown, an embodiment of the present invention provides a job shop scheduling method based on heterogeneous graph neural networks and masked reinforcement learning. This embodiment provides a scheduling optimization example for the TA11 standard dataset (containing 15 jobs and 20 machines) to further illustrate the method of the present invention.
[0038] S1: Read the instance data of the Job Shop Scheduling Problem (JSSP), parse the job, process, and machine information, and obtain a structured instance containing a list of process objects; specifically:
[0039] First, the parameters in the first line of the file are parsed to determine the core scale of 15 jobs and 20 machines. Then, the data is parsed line by line, and each line of information is encapsulated into an Operation object. This object contains core attributes such as job_id (job number), op_id (process number), machine (corresponding processing machine), and proc_time (processing time), and finally forms a standardized list of process objects and a job index mapping table.
[0040] S2: Initialize the JSSPEnv event-driven simulation environment and construct a heterogeneous graph based on the current state. This step specifically includes:
[0041] Construct process nodes and machine nodes, and extract features for each node: extract the processing time of each process and perform min-max normalization; determine the readiness status of the process in real time (0-not ready, 1-ready); use sine / cosine functions (position encoding algorithm) to generate the position codes of the job ID and the machine ID to capture the correlation of category information; the machine node features include the machine release time (i.e., the next available time) and the idle status flag (0-occupied, 1-idle).
[0042] Graph structure construction: Establish two types of edge relationships—first, Precedes edges (process sequence edges, constraining the sequential relationship of processes), connecting adjacent processes within the same job to represent the sequential constraint; second, On edges (equipment association edges, constraining the association relationship between process nodes and machine nodes), connecting processes with their designated processing machines to represent equipment association; finally, encapsulate the node features and edge relationships into HeteroData heterogeneous data objects in PyTorch Geometric format.
[0043] S3: At each decision step, combining the completion status of the prerequisites of the process and the real-time idle status of the machine, an action mask is generated to filter out the set of legal processes that are ready and allowed to start at the current moment; specifically:
[0044] In the decision step function of the JSSPEnv event-driven simulation environment, the `_build_mask` method is called to generate a mask. It iterates through all incomplete operations, checking the `_predecessors_done` attribute (a flag indicating the completion of the preceding operation) to determine if the readiness condition is met. If the operation is in a ready state but has not yet started processing, the corresponding position in the mask is set to 1; otherwise, it is set to 0. If the generated mask vector is entirely 0 (no valid action), the environment automatically calls the `_advance_time` method to advance the time to the next operation completion event, re-triggering the mask generation process. This step masks illegal actions in subsequent decisions, preventing deadlocks or constraint conflicts at the source.
[0045] S4: Inference is performed using a heterogeneous graph neural network. The heterogeneous graph (constructed in step S2) is input into the heterogeneous graph neural network model. The policy network of the heterogeneous graph neural network model outputs the action probability of each process, and the value network of the heterogeneous graph neural network model outputs the current state value assessment V. Specifically:
[0046] In this embodiment, the heterogeneous graph is represented by a HeteroData object, and the heterogeneous graph neural network is a HeteroPolicy model. The HeteroData object is input into the HeteroPolicy model for feature extraction and inference.
[0047] like Figure 3As shown, the HeteroPolicy model includes two layers of HeteroConv convolutional modules (multi-layer heterogeneous graph convolutional layers), with an internal GATConv graph attention layer (using a 4-head attention mechanism, heads=4) to enhance the weights of key features; after each convolutional output, it is processed by ReLU activation function (non-linear activation layer) and LayerNorm layer normalization (layer normalization) to improve the model stability and non-linear expression capability; after two convolutional layers, the deep feature embeddings of the process and the machine are obtained respectively; finally, the output of the decoding layer is used to output the action probability Logits with dimension [N_ops] (total number of processes) through the policy network (including fully connected layers and softmax function), and the current state value evaluation V with dimension [1] through the value network (fully connected layers performing global average pooling).
[0048] S5: As Figure 4 As shown, the heterogeneous graph neural network model is updated and optimized based on the PPO (Proximal Policy Optimization) algorithm:
[0049] Action sampling: The masked_categorical function is called to combine the Logits output by the policy network with the mask generated by S3. The Logits of illegal actions (the value corresponding to the mask is 0) are set to negative infinity to ensure that actions (processes) are sampled only from legal processes. During the training phase, category distribution is used to sample randomly with probability to ensure exploration.
[0050] Trajectory Collection: The agent executes sampling actions and interacts with the JSSPEnv event-driven simulation environment, recording trajectory data for each decision step and storing it in the experience replay pool. Trajectory data includes heterogeneous graph data, action masks, selected actions, and environmental rewards. The reward function aims to minimize the maximum completion time and consists of intermediate and final rewards. Intermediate rewards are constructed based on the relative change of the lower bound of completion time, step penalties, and time advancement penalties. Final rewards are distributed when all processes are completed and are positively correlated with the ratio of the initial lower bound to the actual completion time. Rewards are normalized and used to calculate the advantage function, guiding the policy towards a better schedule.
[0051] Parameter Update: Based on the collected trajectory data, whenever the data batch in the experience revisit pool reaches a set value, the Generalized Dominance Estimation (GAE) algorithm is used to calculate the dominance function. Backpropagation is then performed using the PPO truncation loss formula to synchronously update the network parameters in the policy network, value network, and heterogeneous graph neural network model during reinforcement learning. Iterative optimization continues until the model converges, resulting in a trained scheduling optimization model. In this embodiment, the dominance function aims to maximize scheduling rewards, achieving optimization by minimizing the maximum completion time. An entropy regularization term is also introduced to encourage the agent's exploratory behavior.
[0052] Each action changes the environmental state, but the heterogeneous graph maintains its node types and edge topology, updating only node characteristics in real time: process nodes update their start status, completion status, remaining workload, and associated machine load; machine nodes update their release time, idle status, and remaining load. After each time-advanced action, the environment jumps to the next completion event, updating the process completion status and machine availability, and reconstructing the heterogeneous graph data object for the current state, ready for the next decision step.
[0053] S6: Use a heterogeneous graph neural network model to reason about new scheduling instances and generate the optimal process scheduling sequence.
[0054] In this step, the optimal model parameters saved during training in `best_model.pth` are loaded; the model is switched to evaluation mode, and a greedy strategy (selecting the legal action with the highest probability) is used for inference; the start and end times of all processes are recorded in real time, generating a JSON-formatted scheduling file containing process number, machine number, start and end time, etc.; and a Gantt chart is drawn using the Plotly visualization library (e.g., ...). Figure 5 As shown in the figure, different colors are used to distinguish the processes of each operation, intuitively displaying the load of each machine, the process scheduling sequence, and the final maximum completion time.
[0055] The method of this invention has outstanding structural perception capabilities. It uses heterogeneous graph neural networks to model and schedule states, which can naturally adapt to the heterogeneous attributes of processes and machines, and accurately depict the complex relationship between them. Compared with traditional feature encoding methods, it is more in line with the problem essence of JSSP.
[0056] The method of this invention guarantees constraint satisfaction. It introduces a Petri net-style dynamic masking mechanism to block illegal actions from the decision-making source, ensuring that the model output always meets the process constraints, and greatly improving learning efficiency and the feasibility of scheduling schemes.
[0057] The method of this invention achieves a balance between efficiency and accuracy. Based on the PPO end-to-end learning framework, it realizes an application mode of training once and inference quickly. After training, the solution speed for new scheduling instances far exceeds that of traditional heuristic algorithms, while the scheduling quality is better.
[0058] The preferred embodiments of the present invention have been described in detail above. It should be understood that those skilled in the art can make numerous modifications and variations based on the concept of the present invention without creative effort. Therefore, all technical solutions that can be obtained by those skilled in the art based on the concept of the present invention through logical analysis, reasoning, or limited experimentation on the basis of existing technology should be within the scope of protection defined by the claims.
Claims
1. A job shop scheduling method based on a heterogeneous graph neural network and mask reinforcement learning, characterized in that, Includes the following steps: S1: Read instance data of the job shop scheduling problem, parse job, process and machine information, and construct a structured instance object containing a list of process objects and a job index mapping table; S2: In an event-driven simulation environment, the current scheduling state is extracted and modeled as a heterogeneous graph data object containing process nodes and machine nodes; S3: In each decision step, traverse the unfinished processes and generate the action mask vector of the corresponding process set based on the completion status of the process's predecessor dependencies and the idle status of the specified processing machine. S4: Input the heterogeneous graph data object into the heterogeneous graph neural network model, aggregate node features through the heterogeneous graph convolution mechanism, and output the action probability corresponding to each process and the value assessment scalar of the current state. S5: Combine the action mask vector with the action probability to perform action sampling; use the near-end policy optimization algorithm to interact with the simulation environment to collect trajectory data, and update the network parameters of the heterogeneous graph neural network model with the goal of minimizing the maximum completion time to obtain the scheduling optimization model; S6: Use the scheduling optimization model to reason about the new scheduling instance, and select the operation with the highest probability of action to execute at each step to generate the optimal operation.
2. The job shop scheduling method based on heterogeneous graph neural network and mask reinforcement learning according to claim 1, characterized in that, The process of constructing the heterogeneous graph in step S2 includes: Establish process sequence edges to constrain the order of processes; Establish equipment association edges to constrain the association relationship between process nodes and machine nodes.
3. The job shop scheduling method based on heterogeneous graph neural network and mask reinforcement learning according to claim 1, characterized in that, The process node features include normalized processing time, job number location code, machine number location code, and ready status flag; the machine node features include machine release time and idle status flag.
4. The job shop scheduling method based on heterogeneous graph neural networks and masked reinforcement learning according to claim 3, characterized in that, The feature acquisition process for the process node includes: Extract the processing time of each process and use min-max normalization to process the input feature vector; A location coding algorithm is used to generate location codes for the job number and machine number.
5. The job shop scheduling method based on heterogeneous graph neural networks and masked reinforcement learning according to claim 1, characterized in that, In step S3, during the generation of the action mask vector: Check the completion flag of the preceding process of each process to determine whether it meets the ready condition. If all the preceding processes of an incomplete process have been completed or the process is the first process of the operation and has not started processing, then set the corresponding position in the action mask vector to 1; otherwise, set it to 0. If the action mask vector generated in the current decision step is all 0, it is determined that there is no legal action at the current moment. The simulation environment triggers a priority queue mechanism at the bottom layer, automatically advances the environment time to the moment of the next nearest process completion event, and re-executes state modeling and mask generation.
6. The job shop scheduling method based on heterogeneous graph neural networks and masked reinforcement learning according to claim 1, characterized in that, The heterogeneous graph neural network model includes: Heterogeneous graph convolutional layer: Internally configured graph attention convolutional layers to enhance feature weights. Each convolutional output is connected to a non-linear activation layer and a normalization layer. After several convolutional layers, deep feature embeddings of process nodes and machine nodes are obtained respectively. Decoding layer: Includes policy network and value network. The policy network outputs an action probability distribution with the same dimension as the total number of processes through a fully connected layer and a softmax function. The value network outputs a scalar value assessment of the current state through global average pooling by a fully connected layer.
7. The job shop scheduling method based on heterogeneous graph neural networks and masked reinforcement learning according to claim 1, characterized in that, In step S5, before action sampling, the action probability distribution output by the policy network is combined with the action mask, and the action probability of illegal procedures with corresponding action mask values of 0 is set to negative infinity, ensuring that action sampling is performed only from legal procedures based on the action probability distribution. The intelligent agent performs sampling actions to interact with the simulation environment and collects trajectory data including heterogeneous graph data, action masks, selected actions, and environmental rewards. Based on the collected trajectory data, the generalized advantage estimation algorithm is used to calculate the advantage function, and the backpropagation mechanism of the proximal policy optimization algorithm is used to update the parameters of the policy network, value network and heterogeneous graph neural network in reinforcement learning.
8. The job shop scheduling method based on heterogeneous graph neural networks and masked reinforcement learning according to claim 1, characterized in that, Step S6 also includes generating a JSON-formatted scheduling file containing process number, machine number, and start and end times based on the optimal process; and drawing a Gantt chart using a visualization library.