Workshop scheduling method based on heterogeneous graph neural network and prioritized experience replay

Abstract

This invention discloses a job shop scheduling method based on heterogeneous graph neural networks and priority post-experience replay, relating to the field of dynamic job shop scheduling technology. By loading instances of the dynamic job shop scheduling problem and initializing deep reinforcement learning model parameters, state encoding, policy execution, environmental interaction, and experience storage operations are performed at decision points during training rounds. After each round, synthetic experience is generated through target relabeling, and a hybrid experience replay buffer containing both original and synthetic experience is constructed, with priorities assigned based on temporal difference errors. Experience samples are sampled according to priority, and network parameters are optimized using the PPO algorithm. The optimal model parameters are then loaded to perform real-time scheduling decisions for new instances. This invention improves sample utilization efficiency and policy convergence speed, exhibiting excellent scheduling optimization performance in both static and dynamic environments.

Description

Technical Field

[0001] This invention relates to the field of dynamic job shop scheduling technology, specifically to a job shop scheduling method based on heterogeneous graph neural networks and priority post-experience replay. Background Technology

[0002] As the Dynamic Job Shop Scheduling Problem (DJSP) has become a core combinatorial optimization challenge in manufacturing systems, it has been proven to be an NP-hard problem. Its core objective is to rationally allocate processing machines to multiple jobs and determine the optimal processing sequence under conditions such as process sequence constraints and machine sharing limitations, thereby optimizing key production indicators such as total completion time and equipment utilization. Against the backdrop of modern manufacturing's transformation towards intelligent and flexible operations, the frequent occurrence of dynamic events (such as random job arrivals, machine failures, and uncertain processing times) further exacerbates the scheduling difficulty. In traditional solutions, while exact algorithms can find the optimal solution, they are limited by the curse of dimensionality and are difficult to apply to large-scale dynamic scenarios; heuristic scheduling rules are computationally efficient but suffer from large fluctuations in solution quality and weak generalization ability; metaheuristic methods, while possessing global search capabilities, rely on experience for parameter tuning and have high computational costs, making them difficult to meet real-time scheduling requirements; and early reinforcement learning methods are limited by the size of the state space and can only handle small-scale problems.

[0003] In recent years, Deep Reinforcement Learning (DRL), by incorporating deep neural networks, has achieved effective processing of high-dimensional state and action spaces, providing a new path for solving DJSP (Dynamic Job Shop Scheduling) problems. However, it still faces three major bottlenecks: First, sparse reward signals, as traditional reward mechanisms only provide feedback when the final goal is achieved, resulting in a lack of continuous and effective guidance during the agent's learning process; second, insufficient state modeling, as existing methods struggle to accurately capture the complex topological relationships between heterogeneous entities such as workpieces, processes, and machines in production systems, requiring extensive manual feature engineering; and third, low sample efficiency, as experience replay mechanisms fail to fully utilize successful and failed experiences, leading to slow policy convergence and a tendency to get trapped in local optima. These problems severely restrict the application effectiveness and widespread value of DRL methods in real-world dynamic job shop scheduling scenarios.

[0004] Chinese patent (CN113469491A) proposes a flexible shop floor scheduling method based on reinforcement learning and graph neural networks. Addressing the difficulty of modeling complex entity relationships in traditional methods, it innovatively summarizes shop floor information into a flexible shop floor graph model containing three types of entities: equipment, processes, and operations. High-dimensional features are extracted through multi-layer computation of the graph neural network, and the policy network is optimized based on the REINFORCE algorithm, significantly improving the computational efficiency and scalability of scheduling decisions. This patent effectively captures the constraints between operations and the dynamic state of equipment through collaborative modeling of operation embedding, order embedding, and equipment embedding, alleviating to some extent the dependence of traditional methods on manual features. However, it still has significant shortcomings: firstly, the experience replay uses a simple random sampling method, failing to distinguish the learning value of experience samples, resulting in low sample utilization and limited policy convergence speed; secondly, the reward mechanism design does not fully consider the sparse reward problem in dynamic scenarios, lacking effective feedback for intermediate scheduling steps, making it difficult for the agent to quickly learn robust scheduling strategies; and thirdly, the graph model only focuses on static entity relationships, lacking adaptability to dynamic events (such as processing time fluctuations and sudden machine failures).

[0005] To address the technical shortcomings of patent CN113469491A, Chinese patent (CN120742806A) proposes a flexible job shop scheduling method based on graph neural networks and deep reinforcement learning. This method introduces heterogeneous graph modeling to depict the complex relationships between jobs and machines, and innovatively designs a meta-relationship-based heterogeneous graph neural network (MRHGNN). Differentiated graph convolution operations are used for feature embedding based on different meta-relationships, enhancing the ability to capture global information. This patent optimizes feature extraction accuracy through the synergistic effect of GAT and Graph Transformer, and designs an improved reward mechanism that directly quantifies completion time, further enhancing the accuracy of scheduling decisions. However, the scheme still has limitations: First, it does not solve the problem of uneven quality of experience samples and does not introduce a priority mechanism to screen high-value experience, resulting in redundant samples diluting the learning effect; Second, although the reward mechanism focuses on optimizing completion time, it does not combine key indicators such as process waiting time and equipment load balancing, and the feedback guidance is not comprehensive enough; Third, it does not adequately alleviate the problem of reward sparsity in dynamic scenarios and does not generate supplementary experience through mechanisms such as target relabeling, so the agent is still prone to learning stagnation in complex dynamic environments.

[0006] Therefore, there is an urgent need for a dynamic job shop scheduling method that can simultaneously solve the three core problems of inaccurate state modeling, sparse rewards, and low sample efficiency. Summary of the Invention

[0007] To address the aforementioned technical problems, this application discloses a workshop scheduling method based on heterogeneous graph neural networks and priority post-experience replay, specifically including:

[0008] Load an instance of the dynamic job shop scheduling problem, the instance including a set of workpieces, a set of machines, a sequence of operations, a processing time matrix, and dynamic event settings; initialize the parameters of the deep reinforcement learning model, the model parameters including the weights of the heterogeneous graph neural network encoder, the Actor network, and the Critic network.

[0009] In each training round, the scheduling environment is reset to its initial state, and at each decision time... The state coding, policy execution, environment interaction, and experience storage operations are executed sequentially.

[0010] After the round ends, the post-experience replay engine is activated. For each original experience collected in the round, a target remarking strategy is used to generate synthetic experience, and the synthetic experience is stored in the HER experience buffer.

[0011] A hybrid experience replay buffer is constructed, which is obtained by merging the original experience buffer and the HER experience buffer. A priority is calculated and maintained for each experience in the hybrid experience replay buffer, and the initial value of the priority is determined based on the absolute value of the time-series difference error.

[0012] In each model update step, a small batch of experience samples is sampled from the hybrid experience replay buffer according to the priority and a preset probability. The total loss function of the PPO algorithm is calculated using the sampled experience batches, and the parameters of the Actor network and Critic network are optimized by gradient descent.

[0013] Load the optimal model parameters after training, perform real-time scheduling decisions on new dynamic job shop scheduling instances, generate a complete scheduling scheme, and output the final completion time.

[0014] Preferably, the status coding includes:

[0015] At the moment of decision The current scheduling environment state is constructed as a heterogeneous graph. , where the node set Includes process nodes and machine nodes edge set It includes three types of semantic edges: process sequence edges, compatibility edges, and occupancy edges. In dynamic job shop scheduling, the set of all processes that are to be processed or are being processed in the system at decision time $t$ is the process set.

[0016] Configure time-varying attributes for the process node, including process ready status, remaining processing time, cumulative waiting time, and a binarized completion indicator; configure status attributes for the machine node, including machine availability status and remaining busy time.

[0017] Heterogeneous graphs carrying node attributes The input is fed into a heterogeneous graph neural network encoder, which extracts an end-to-end feature representation of the current scheduling environment state through node embedding and information propagation in the graph neural network.

[0018] Preferably, the edge set The definitions and functions of the three types of semantic edges are as follows:

[0019] The process sequence edge is a directed edge that uniquely connects adjacent process nodes within the same operation, and is used to explicitly depict the process sequence constraints of the operation.

[0020] The compatibility edge is an undirected edge that connects each process node to all its selectable machine nodes, representing the potential processing location selection of the process. The compatibility edge is also configured with a binary feasibility indicator to indicate whether the corresponding machine can be assigned to the process.

[0021] The occupied edge is an undirected edge that connects the machine node to the process node it is currently processing in real time, and is used to intuitively identify the real-time processing relationship between the machine and the process.

[0022] Preferably, the strategy execution includes:

[0023] In the current scheduling state Below, based on heterogeneous graphs Dynamically construct a set of actionable actions based on node attributes and edge set constraints. The set of actionable actions consists of all valid process-machine assignment pairs, satisfying the constraint that ready processes are matched with idle and compatible machines.

[0024] The heterogeneous graph The feature representation and the specified process completion target The inputs are fed into the Actor network, which calculates an unnormalized action score for each action in the set of possible actions. All action scores are then normalized using the softmax function to form an action probability distribution.

[0025] Scheduling actions are obtained by random sampling based on the action probability distribution. The scheduling action is a unique process-machine allocation instruction.

[0026] Preferably, each experience in the hybrid experience playback buffer priority and the preset sampling probability The calculation method is as follows:

[0027] ,in For experience The timing difference error, A preset non-zero positive constant is used to ensure that all experiences have a non-zero sampling probability;

[0028] ,in To control the hyperparameters of priority sampling intensity, The total number of experience samples currently stored in the mixed experience replay buffer. For the first The original priority of each empirical sample For the first The result of exponentially transforming the original priorities of a number of experience samples.

[0029] Preferably, the timing difference error The calculation formula is: ,in For decision-making steps Instant rewards received This is a reward discount factor used to quantify the impact of future rewards on current decisions. This is the state-object pair value estimate output by the Critic network. For the parameters of the Critic network, To achieve the goal of completing the process based on original experience, To complete the target for the new process after the target has been remarked.

[0030] Preferably, the instant reward The target conditional reward function is calculated, and the specific calculation process of the target conditional reward function is as follows:

[0031] Calculate the decision time Total waiting time for ready processes , ,in For the first A set of processes that are ready but not yet assigned. For process At any moment The cumulative waiting time;

[0032] Then, calculate the immediate reward based on the completion status of the process. ,in A fixed positive reward upon completion of the process. The weighting coefficient for the waiting time penalty term. Let be the target indicator function, if The specified process is in status If completed in the middle If not completed, the value is 0.

[0033] Preferably, the total loss function of the PPO algorithm is: ,in Indicates time step Calculate the mathematical expectation from the empirical sample. and These are preset hyperparameters used to balance the weights of the value function loss and the entropy regularization term. For the network parameters to be optimized, For strategic losses, For the value function loss, This is the entropy regularization term.

[0034] Preferably, the strategy loss and value function loss The calculation formulas are as follows: ,in The ratio of the probabilities of the new and old strategies is given. For the new strategy in the state Select Action The probability, The probability of the old strategy. For the dominant function, For the cropping operation, The threshold for the cropping operation; ,in For target value, For the valuation of the old Critic network, Hyperparameters for value tailoring.

[0035] Preferably, the real-time scheduling decision includes:

[0036] At each decision point in the operation of the scheduling system, the current state of the scheduling environment is perceived and constructed into a heterogeneous graph according to the state coding method. The state feature representation is extracted by the heterogeneous graph neural network encoder;

[0037] The state feature representation is input into the trained Actor network, and a deterministic strategy is used to select the action with the highest probability from the Actor network's output action probability distribution. The action Assign instructions to the optimal process-machine configuration in the current state;

[0038] The action The command is sent to the actual production system or simulation environment for execution. The scheduling environment automatically evolves to the next state based on the execution result and real-time dynamic events. ;

[0039] Repeat the above steps, making scheduling decisions based on the latest status at each new decision point, until all workpieces are processed or the scheduling period ends, generating a complete scheduling Gantt chart and process assignment list.

[0040] Compared with the prior art, the technical solution of this application has the following technical effects:

[0041] This invention constructs a heterogeneous graph containing process nodes, machine nodes, and multiple semantic edges to achieve a structured representation of the scheduling environment. It naturally integrates the features and relationships of heterogeneous entities, eliminating the need for manual feature engineering. Through end-to-end extraction of state features via graph neural networks, it accurately captures the complex topological relationships and dynamic changes of the production system, providing comprehensive and accurate input support for scheduling decisions.

[0042] The hybrid experience replay mechanism of this invention generates synthetic experience through target relabeling and performs priority sampling by combining temporal difference error. This not only enriches the learning samples but also highlights the role of high-value experience, effectively alleviates the reward sparsity problem, significantly improves sample utilization efficiency, accelerates policy convergence, and enables the agent to learn stable and efficient scheduling policies more quickly.

[0043] The multi-component objective conditional reward function designed in this invention decomposes the terminal reward into local signals and provides dense and well-guided learning feedback through positive rewards for process completion and penalties for waiting time. This guides the agent to optimize towards the goal of minimizing the total completion time, while adapting to dynamic scenarios with uncertain processing times, thus enhancing the practicality and adaptability of the scheduling strategy.

[0044] The complete deep reinforcement learning scheduling framework constructed in this invention optimizes network parameters through the PPO algorithm during the training phase and outputs the optimal process-machine allocation instructions using a deterministic strategy during the inference phase. It has excellent robustness and rescheduling capability, and can efficiently generate scheduling schemes in static and dynamic uncertain environments, ensuring the smooth and efficient production process.

[0045] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the preferred embodiments of this application are described in detail below with reference to the accompanying drawings.

[0046] The above and other objects, advantages and features of this application will become more apparent to those skilled in the art from the following detailed description of specific embodiments in conjunction with the accompanying drawings. Attached Figure Description

[0047] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In all drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0048] Based on the description of the figures and their corresponding technical content in the document, the titles of the figures are as follows:

[0049] Figure 1 Flowchart of the workshop scheduling method based on heterogeneous graph neural network and priority post-event experience replay;

[0050] Figure 2 A schematic diagram of the heterogeneous graph representation of dynamic job shop scheduling and the Actor-Critic strategy learning framework;

[0051] Figure 3 A schematic diagram of the HER target remarking and priority sampling process in priority post-experience replay;

[0052] Figure 4 Comparison of empirical sampling probability distribution curves under different priority indices α;

[0053] Figure 5 A comparison curve of training performance with and without HER mechanism under high processing time uncertainty;

[0054] Figure 6 A comparison curve of total completion time for dynamically scheduled instances under a normal distribution;

[0055] Figure 7 A comparison curve of the total completion time of dynamic scheduling instances under uniform distribution. Detailed Implementation

[0056] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. In the following description, specific details such as specific configurations and components are provided merely to help fully understand the embodiments of this application. Therefore, those skilled in the art should understand that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. In addition, for clarity and brevity, descriptions of known functions and structures are omitted in the embodiments.

[0057] It should be understood that the phrase "an embodiment" or "this embodiment" throughout the specification means that a specific feature, structure, or characteristic related to the embodiment is included in at least one embodiment of this application. Therefore, "an embodiment" or "this embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0058] Furthermore, reference numerals and / or letters may be repeated in different examples within this application. Such repetition is for the purpose of simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or settings discussed.

[0059] In this article, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, B exists alone, and A and B exist simultaneously. The term " / and" in this article describes another type of relationship between related objects, indicating that two relationships can exist. For example, A / and B can mean: A exists alone, and A and B exist alone. In addition, the character " / " in this article generally indicates that the related objects before and after it are in an "or" relationship.

[0060] In this article, the term "at least one" is merely a description of the relationship between related objects, indicating that there can be three relationships. For example, "at least one of A and B" can mean: A exists alone, A and B exist simultaneously, or B exists alone.

[0061] It should also be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion.

[0062] Example 1

[0063] This embodiment mainly describes a workshop scheduling method based on heterogeneous graph neural networks and priority post-event experience replay, such as Figure 1 As shown, it specifically includes:

[0064] Load an instance of the dynamic job shop scheduling problem, the instance including a set of workpieces, a set of machines, a sequence of operations, a processing time matrix, and dynamic event settings; initialize the parameters of the deep reinforcement learning model, the model parameters including the weights of the heterogeneous graph neural network encoder, the Actor network, and the Critic network.

[0065] In each training round, the scheduling environment is reset to its initial state, and at each decision time... The state coding, policy execution, environment interaction, and experience storage operations are executed sequentially.

[0066] After the round ends, the post-experience replay engine is activated. For each original experience collected in the round, a target remarking strategy is used to generate synthetic experience, and the synthetic experience is stored in the HER experience buffer.

[0067] A hybrid experience replay buffer is constructed, which is obtained by merging the original experience buffer and the HER experience buffer. A priority is calculated and maintained for each experience in the hybrid experience replay buffer, and the initial value of the priority is determined based on the absolute value of the time-series difference error.

[0068] In each model update step, a small batch of experience samples is sampled from the hybrid experience replay buffer according to the priority and a preset probability. The total loss function of the PPO algorithm is calculated using the sampled experience batches, and the parameters of the Actor network and Critic network are optimized by gradient descent.

[0069] Load the optimal model parameters after training, perform real-time scheduling decisions on new dynamic job shop scheduling instances, generate a complete scheduling scheme, and output the final completion time.

[0070] Furthermore, an instance of the dynamic job shop scheduling problem can be defined within a continuous time domain, where the production system consists of a set of jobs. Composition, in which each job It includes a series of orderly processes. Each process step can only begin after all its preceding processes are completed to satisfy process sequence constraints, and can be assigned to any machine in its compatible set for processing. This represents the total number of processes in the system. This approach uses uncertain processing time as a dynamic event and aims to minimize the total completion time as the modeling guideline.

[0071] An end-to-end scheduling paradigm based on deep reinforcement learning is adopted, modeling the scheduling process as a Markov decision process. This allows the agent to learn the optimal scheduling strategy directly from the high-dimensional, dynamic system state through continuous interaction with the environment, without relying on any predefined heuristic rules. To achieve a unified and unambiguous representation of scheduling elements among states, actions, and objectives, all procedures are assigned a unique global index, specifically through the introduction of bijective mapping. ,in Indicate homework The One process, The corresponding global process index provides a compact and consistent representation for each process, which is used to define scheduling status, action feasibility, and process-level optimization goals.

[0072] state Used to capture decision moments Complete dynamic environmental information is represented using heterogeneous graphs. , where the node set Includes process nodes and machine nodes edge set It includes connecting arcs representing pre-order constraints between processes, and edges indicating the allocation relationship between processes and their compatible machines. Dynamic attributes are embedded in each node, such as process waiting time and delivery urgency, as well as machine status (idle, busy, faulty) and queue length. This graph-based representation provides a comprehensive and structured description of the scheduling situation at any given time.

[0073] In the action space, each executable action corresponds to a feasible scheduling decision that can be implemented when the system is in a specific state. This action space adopts a dynamic construction mechanism with composite structural characteristics, and its basic components consist of schedulable procedures and their corresponding available machines. Specifically, in the state... The following are feasible actions. This corresponds to assigning a ready operation (all preceding operations have been completed) to an idle and compatible machine, thus the motion space... Dynamically constructed as all valid operation-machine pairs, its expression is: This mechanism ensures the feasibility of actions and aligns naturally with the event-driven scheduling paradigm.

[0074] Furthermore, to effectively guide deep reinforcement learning agents toward the global optimization objective of minimizing the total completion time (makespan), this paper designs a dense and multi-component objective conditional reward function. This function decomposes the originally highly sparse terminal reward into several localized signals tightly coupled with the system's dynamic state, thereby providing continuous, differentiable, and semantically clear learning feedback during the decision-making process. Overall Reward The calculation formula is as follows: First, calculate the decision time. Total waiting time for ready processes ,in Indicates the first A set of processes that are ready and can be scheduled at any given time. Indicate process At any moment The cumulative waiting time; then, the instant reward is calculated based on the completion status of the process. ,in This indicates the re-marked target, corresponding to the completion of a specific process. It is a fixed reward given upon completion of the process. The weighting coefficient for the waiting time penalty term. Let be the target indicator function, if The specified process is in status If it has been completed Otherwise, the reward function is 0. It is worth noting that this reward function only depends on observable state transitions (especially process completion events) and does not depend on any prior known processing time information, ensuring that the reward calculation is still valid when the processing time is uncertain, and is fully compatible with the stochastic dynamic state transition characteristics of the proposed Markov decision process.

[0075] State representation is a core element of scheduling environment modeling, and its formal definition directly determines the structural dimension of feature expression and the scale of state space. This paper presents a state representation of dynamic job shop scheduling based on heterogeneous graphs, and makes corresponding improvements to the heterogeneous graphs to address the event of sudden machine failure.

[0076] At every decision moment System status A Markov snapshot of the current scheduling environment is provided, and the state is represented as a heterogeneous graph. , where the set of nodes It contains two types of nodes: process nodes and machine nodes, corresponding to processes and processing resources with global indices, respectively. In the implementation, these attributes are encoded as real-valued feature vectors attached to each node and updated at each decision time, thus forming the node feature matrix input to the heterogeneous graph neural network. Node set In the middle, process nodes Associated with a series of time-varying attributes, including ready state, remaining processing time, cumulative waiting time, and a binarized completion indicator; machine node Then it encodes its availability status and remaining busy time. Edge set It includes three semantically defined edge types to characterize all relationships: directed process sequence edges connect adjacent processes within the same job, used to characterize process sequence constraints; compatibility edges connect each process to all its available machines, representing the potential processing position selection of the process, and may additionally include a binary feasibility indicator to indicate whether the corresponding machine is currently available to be assigned to the process; occupancy edges connect machine nodes to the process nodes they are currently processing, visually indicating which machine is operating which process. These edges are considered typed structural connections to define the information propagation pattern in the graph, where process sequence edges explicitly encode process order constraints in the technology.

[0077] Furthermore, the processing time of each operation is modeled as a random variable. The initial estimated processing time of each operation is set to the nominal processing duration provided by its scheduling instance, which corresponds to the expected value of the underlying random processing time distribution. This expected value is fixed at the beginning of each episode and is not updated during the execution. Due to the uncertainty of processing time, the actual processing duration of an operation cannot be known at the decision-making moment. During the execution, the agent can only observe the elapsed processing time and an estimated remaining processing time. In the specific implementation, the remaining processing time is not based on the actual time required to complete, but is calculated based on the expected processing duration associated with each operation. Specifically, the estimated remaining time is obtained by subtracting the elapsed processing time from the expected processing duration, and it is set to zero when the result is less than zero.

[0078] The ultimate optimization objectives of DJSP are global and terminal metrics, which can only be accurately calculated at the end of the entire scheduling period (when all jobs are completed). This results in the agent only receiving a non-zero reward signal at the last step in a long decision-making sequence. At each step before that, without a carefully designed intermediate reward, the immediate reward is either zero or a constant that does not contain target information. The agent cannot effectively learn from such sparse feedback which intermediate decisions are beneficial to the final goal, thus leading to the problem of sparse rewards. The goal-conditioned reinforcement learning framework is adopted, and the off-policy experience replay mechanism is integrated into the proposed scheduling model.

[0079] Furthermore, the policy function and the value function not only condition on the current system state but also explicitly depend on a well-defined scheduling goal . In dynamic job shop scheduling, a goal is defined as the completion event of a specific operation that can be achieved within the current scheduling episode. Formally, each goal in the goal set corresponds to the completion of a specified operation. According to the global operation indexing scheme introduced earlier, each operation is identified by a unique integer index . Therefore, in the implementation, the goal is represented as a scalar - namely, the global index of this operation, which directly points to a specific operation node in the heterogeneous graph. This goal definition avoids using abstract or heuristic optimization metrics (such as machine utilization ratio) and ensures that whether a goal is achieved can be directly inferred from the state transition . Specifically, each operation node maintains a binary completion indicator as its node attribute. For a given goal , whether it is achieved can be checked by examining the graph state from Transferred to During the process, process nodes The completion flag is changed from 0 to 1 to verify the completion status. Since the completion status and waiting time of the process are encoded as node attributes in the heterogeneous graph, the reward can be calculated deterministically from the graph state transition. This design establishes a direct correspondence between goal achievement and reward generation, laying a key foundation for the effective application of the HER mechanism.

[0080] Furthermore, in the proposed objective-conditional reinforcement learning, scheduling decisions are generated by an Actor-Critic architecture, whose policy network and value network are respectively derived from parameters. and Parameterization. Policy network. At the same time, based on the current scheduling state Complete the target with a specified process This is a condition for achieving target-aware decision-making under sparse reward signals. Specifically, at each decision time... Actor networks are in heterogeneous graph state and conditional objectives As input, the set of actionable actions dynamically constructed in the current state (i.e., all valid process-machine assignment pairs), the Actor calculates an unnormalized action score for each actionable action, and these scores are then processed in... The above is normalized using the softmax function to form a probability distribution, and the scheduling action is sampled based on this distribution. Critic Network Use the same input as the Actor (i.e., graph state) and target It outputs a scalar value to estimate the expected discounted return from the current state-target pair. This value estimate is used to subsequently calculate the temporal difference (TD) error and drive policy updates and optimizations within the PPO framework.

[0081] Furthermore, such as Figure 3 As shown, HER addresses the sparse reward challenge through a goal relabeling strategy, reconstructing the original failed trajectory for the unachieved goal into a successful experience relative to the achieved state, thus significantly improving the agent's generalization ability in multi-objective tasks. However, although HER can generate rich alternative experiences, it does not distinguish the quality of experience, introducing a large number of suboptimal or redundant samples, thereby diluting the overall quality of the experience replay buffer. Therefore, this scheme improves HER. The core of HER lies in reinterpreting the same trajectory based on different goals. The collected experience tuples HER uses remapping to assign a new target to it. New Goals Sample from the final state achieved in the current round, i.e. The target relabeling process can be formally represented as: ,in Indicates that according to the new target The reward signal is recalculated, and this process incorporates the experience of previous failures. Transformed relative to the new target Successful experience This generates additional learning samples with positive rewards.

[0082] Furthermore, directly applying these regenerated samples would overfit a large number of simple or suboptimal experiences, thus requiring priority sampling. This scheme uses each empirical tuple... Assign a priority This allows for non-uniform sampling based on this priority, thus overcoming the limitations of uniform sampling. Priority Based on the absolute value setting of the TD error, its calculation formula is as follows: ,in , Representing experience The timing difference (TD) error, It is a very small positive constant used to ensure that the sampling probability remains non-zero for all experiences. In the decision-making step The immediate rewards obtained (including rewards from the original experience, as well as rewards recalculated after the experience is re-marked by the HER mechanism). Discount factor, value function Given a parameter The criteria network is obtained by performing one forward propagation. The sampling probability formula based on this priority is: ,in It is a hyperparameter that controls the priority intensity. This represents the total number of experience samples currently stored in the experience replay pool. Indicates the first The original priority of each empirical sample Perform an exponential transformation.

[0083] Furthermore, the raw experiences generated through the interaction between the agent and the environment, as well as the post-event experiences generated by the HER mechanism, will be jointly stored in a unified experience replay buffer. For each newly deposited experience, its priority is initialized using the absolute value of its initial TD error. Experience with HER generation Its initial TD error is based on the current value network. Conducting an evaluation ensures that the results are consistent with alternative targets. Experiences with high learning value can be promptly assigned higher sampling priority. In each parameter update step, the calculated sampling probability is used... From the mixed experience buffer A small batch of experiences is sampled, which are then used to calculate the loss function and update the network parameters, and their priority in the replay buffer is updated accordingly. This ensures the dynamic and timely maintenance of experience priority, so that it can accurately reflect the current relative learning value of the experience.

[0084] Furthermore, when facing a DJSP, the model training process is as follows: First, load the dynamic job shop scheduling problem instance, including the set of jobs, machines, process sequence, processing time matrix, and dynamic events. Initialize the deep reinforcement learning model parameters, including the weights of the heterogeneous graph neural network encoder, Actor network, and Critic network. In each training round, the environment is reset to its initial state, and at each decision time... The agent performs state encoding, policy execution, environmental interaction, and experience storage operations. At the end of a round, it activates the post-event experience replay engine, uses a target relabeling policy to store all synthetic experiences into the HER experience buffer for each original experience collected during the round, and constructs a hybrid experience replay buffer. For each of these experiences Calculate and maintain a priority The initial value is the absolute value of its timing difference error. In each model update step, based on priority and probability... From the buffer A small batch of empirical samples is sampled, and the total loss function of the PPO algorithm is calculated using the sampled empirical batch. The parameters of the policy network (Actor) and value network (Critic) are optimized through gradient descent.

[0085] The formula for calculating the total loss function is as follows: ,in Indicates time step Calculate the expectation from the empirical sample. and These are hyperparameters, used to balance the weights of the value function loss and the entropy regularization term. This represents the network parameters to be optimized. Policy loss. This is the core of PPO, which ensures training stability by limiting the policy update magnitude. Its formula is as follows: ,in , , Is the new strategy in the state? Select Action The probability, It is the probability of the old strategy. It is a timing difference error. It is a discount factor. It's a hyperparameter. Indicates the cropping operation and ensures Restricted to the range Inside.

[0086] Value function loss Used to train the Critic network to make its estimation of state value more accurate, its calculation formula is as follows: ,in , This is a value estimate of the old Critic network. It is a hyperparameter for value pruning. Entropy regularization term. Used to encourage strategy exploration and prevent premature convergence to a local optimum, its calculation formula is: ,in , The strategy is distributed across the states. The information entropy is such that the larger the entropy value, the more uniform the distribution of actions of the strategy in a given state, and the stronger the exploratory nature.

[0087] In the inference phase, the trained model is used to make real-time or near-real-time scheduling decisions for new and unseen dynamic job shop scheduling instances. The specific steps are as follows: Load the optimal model parameters that have converged and been saved during the training phase; initialize the heterogeneous graph neural network encoder, Actor network, and Critic network; load or receive instance information of the dynamic job shop scheduling problem to be solved; initialize the simulation environment; at each decision point in the scheduling system's operation, the system perceives the current environmental state and constructs it as a heterogeneous graph. ;State diagram The loaded heterogeneous graph neural network encoder is input to extract the feature representation of the current state. This state feature is then input into the pre-trained Actor network. Unlike the sampling during the training phase, the inference phase adopts a deterministic strategy, directly selecting the action with the highest probability from the probability distribution output by the Actor network. This action corresponds to the optimal scheduling instruction for assigning a specific process to a specific machine in the current state; the decision action... The command is issued to the actual production system or simulation environment for execution; the environment automatically evolves to the next state based on the execution result and possible dynamic events. Repeat the above steps to make a scheduling decision based on the latest state at each new decision point; this process continues until all workpieces are processed or the end of the scheduling period is reached, thereby generating a complete scheduling plan; record all state-action pair sequences generated throughout the reasoning process, i.e., a complete scheduling Gantt chart or work assignment list; output the final completion time for production management personnel to evaluate and use.

[0088] This implementation achieves end-to-end feature learning by accurately modeling the features and relationships of heterogeneous entities in the scheduling environment using heterogeneous graphs, thus avoiding manual feature engineering. The hybrid experience replay mechanism integrates post-experience relabeling and priority sampling, effectively alleviating the reward sparsity problem and improving sample utilization and policy convergence speed. Combined with a multi-component objective conditional reward function to provide dense learning feedback, and combined with the PPO algorithm to optimize network parameters, the scheduling system can significantly shorten the total completion time in both static and dynamic uncertain environments, while also possessing excellent robustness and real-time rescheduling capabilities.

[0089] The experiment uses a dual-validation dataset constructed with multi-scale standard benchmark instances and extended dynamic test instances. The standard benchmark instances use the classic DJSP dataset, covering instances with different combinations of workpiece and machine numbers, such as 10×5, 15×5, 20×5, 10×10, 15×10, 20×10, 30×10, 15×15, 20×20, 30×15, 30×20, and 50×15 (e.g., la01-la40, ta21-ta52), covering typical production scenarios from small to large scale, to verify scheduling optimization performance in static scenarios. The extended dynamic test instances are generated based on three representative standard benchmark instances, introducing the randomness of process processing time as the core dynamic event. Two probability distributions (normal distribution and uniform distribution), four uncertain process proportions (25%, 50%, 75%, and 100%), and three disturbance levels are designed to construct a total of 72 enhanced test instances to simulate complex scenarios such as processing time fluctuations and process uncertainty in actual production, and to verify robustness and adaptability under dynamic uncertainty environments. To ensure fairness and effectiveness of the experiment, random perturbations were introduced into the processing time of all instances while maintaining a constant average processing time. This avoided result bias caused by differences in processing time baselines. Simultaneously, the sampling probability changes under different empirical quantities were recorded during the experiment. Figure 4 As shown, the sampling probability for different α values ​​increases with the increase in the number of experiences. The green line with α=1 shows the most significant increase, followed by the red line with α=0.4. The black line with α=0 remains at a low level and increases slowly. This intuitively reflects the priority mechanism's tendency to sample high-value experiences differently, and also provides basic data support for the effectiveness verification of the subsequent mixed experience playback mechanism.

[0090] The experiment included multiple comparison groups, including traditional Priority Drill-Down (PDR) scheduling rules, existing deep reinforcement learning scheduling methods, and variant models of the proposed framework (HGNN+PER uses only priority experience replay, and HGNN+HER uses only post-experience replay). For PDR, the optimal results from SPT, MWKR, LRM, FIFO, LIFO, and LPT were selected as the benchmark to ensure comprehensiveness and representativeness of the comparison. Experimental parameters were rigorously optimized: the priority index α was set to 0.4 to balance priority sampling and sample diversity, avoiding over-biasing towards high-value experiences that could lead to policy overfitting; all models were trained independently three times, and the average results were recorded to reduce random error and improve the reliability of the conclusions; the PPO algorithm was used during training to optimize network parameters and hyperparameters. The optimal value was determined through grid search to ensure that all comparison methods were at their best performance, avoiding the impact of parameter tuning differences on the fairness of the results. During the parameter tuning process, the influence of the α value on the sampling probability was observed in detail, further confirming the rationality of α=0.4 in balancing sampling efficiency and sample diversity, and also laying the parameter foundation for subsequent training performance comparison experiments.

[0091] The experimental results in static scenarios are shown in the table below. The proposed framework (Ours) demonstrates excellent scheduling performance in standard instances of different sizes: In the 10×5 la01 instance, the total completion time reaches 666, which is on par with existing deep reinforcement learning scheduling methods and better than HGNN+PER (676) and HGNN+HER (669); In the 10×10 la16 instance, the total completion time is as low as 990, which is significantly better than Min PDR (1040), existing deep reinforcement learning scheduling methods (1027) and two variant models, demonstrating the optimization advantage for medium-sized scheduling problems; In the 15×10 la22-la25 instances, the total completion times are 967, 1032, 973 and 1042, respectively, which are the best values ​​among all the comparison methods, indicating that the framework can still accurately capture the scheduling logic under complex constraints of multiple artifacts and multiple machines; In larger-scale 20×10, 30×10 and 50×15 instances, the total completion time is 967, 1032, 973 and 1042, respectively, which are the best values ​​among all the comparison methods, indicating that the framework can still accurately capture the scheduling logic under complex constraints of multiple artifacts and multiple machines; In the examples, the proposed framework maintains its advantages. For instance, in the 50×15 ta52 instance, the total completion time is only 2961, further shortening the time compared to existing deep reinforcement learning scheduling methods (3056). This fully demonstrates its optimization effect on total completion time in static scenarios. Moreover, the performance advantage does not significantly diminish as the problem size increases, exhibiting good scalability. The overall comparison trend of static instances shows that the completion time of the proposed framework is consistently lower than Min PDR and existing deep reinforcement learning scheduling methods, with a more pronounced advantage in large-scale instances. This further confirms the synergistic effect of heterogeneous graph modeling and hybrid experience replay mechanisms in static scheduling optimization.

[0092] scale Example Min PDR Wu HGNN+PER HGNN+HER Ours 10×5 la01 683 666 676 669 666 15×5 la06 929 926 931 929 926 20×5 la11 1270 1222 1240 1239 1222 10×10 la16 1040 1027 1040 1039 990 15×10 la21 1236 1193 1189 1193 1124 15×10 la22 1071 997 1034 1029 967 15×10 la23 1159 1039 1109 1098 1032 15×10 la24 1085 1015 1054 1049 973 15×10 la25 1112 1050 1088 1082 1042 20×10 la26 1392 1283 1320 1315 1259 20×10 la27 1410 1353 1388 1388 1341 20×10 la28 1473 1305 1402 1389 1266 20×10 la29 1288 1288 1315 1309 1244 20×10 la30 1537 1395 1456 1449 1368 30×10 la31 1820 1786 1239 1246 1784 30×10 la32 1889 1850 1239 1246 1850 30×10 la33 1794 1720 1762 1758 1719 30×10 la34 1855 1746 1784 1779 1721 30×10 la35 2046 1896 1948 1941 1897 15×15 la36 1394 1357 1239 1246 1368 15×15 la37 1582 1511 1239 1246 1482 15×15 la38 1358 1327 1345 1351 1294 15×15 la39 1399 1343 1382 1368 1333 15×15 la40 1352 1314 1326 1334 1312 20×20 ta21 1978 1876 1901 1918 1824 20×20 ta22 1881 1794 1843 1843 1739 30×15 ta31 2156 1965 1988 1993 1927 30×15 ta32 2173 2096 2126 2133 2047 30×20 ta41 2514 2398 2451 2444 2281 30×20 ta42 2419 2305 2364 2351 2253 50×15 ta51 3461 3155 3215 3198 3136 50×15 ta52 3282 3056 3148 3144 2961

[0093] like Figure 5As shown, the training performance comparison under dynamic high uncertainty scenarios reveals that the proposed framework with the HER mechanism converges significantly faster than HGNN+PER, and ultimately achieves a lower average completion time, demonstrating the effectiveness of post-experience replay in alleviating the reward sparsity problem. During training, training performance comparison curves are plotted with training epochs on the horizontal axis and average completion time on the vertical axis. The red line marked with HER and the black line without HER show a clear divergence: In the early stages of training, both curves begin to decline from similar high completion times. The black line without HER declines gradually, maintaining an average completion time of around 1270 at the end of 2000 training epochs; while the red line with HER declines extremely rapidly, with the average completion time dropping below 1200 at 400 epochs, further decreasing to around 1150 at 800 epochs, and stabilizing at a low level of around 1090 at the end of 2000 training epochs, with even smaller fluctuations. This curve change intuitively reflects that the HER mechanism generates synthetic experience through target relabeling, supplementing a large number of positive reward samples, enabling the agent to quickly learn effective scheduling logic in the early stages of training. It also shows that the hybrid experience replay mechanism further strengthens this advantage, allowing the policy to converge to a better local optimum, avoiding overfitting or convergence stagnation caused by a single mechanism.

[0094] like Figure 6 , 7 As shown, the dynamic scheduling results under a normal distribution, with instances on the horizontal axis and total completion time on the vertical axis, present a clear comparative trend. The black line representing Min PDR, the red line representing existing deep reinforcement learning scheduling methods, and the blue line representing the proposed framework exhibit an overall top-bottom-lowest distribution: in most instances, the black line representing Min PDR is consistently at the highest position, followed by the red line representing existing deep reinforcement learning scheduling methods, while the blue line representing the proposed framework is consistently at the lowest position. Furthermore, significant peak completion times appear in some instances, with Min PDR's peak exceeding 3000, existing deep reinforcement learning scheduling methods approaching 3000, while the proposed framework's peak is only around 2900. Moreover, the decline rate after the peak is faster, resulting in a significantly lower overall completion time compared to the other two methods, with a smaller fluctuation range. This trend indicates that in a normally distributed dynamic environment, the proposed framework, by accurately capturing the dynamic relationship between processes and machines through heterogeneous graphs and combining a priority post-experience replay mechanism, effectively addresses processing time fluctuations, significantly reduces total completion time, and demonstrates greater stability.

[0095] The dynamic scheduling results under uniform distribution are consistent with the normal distribution trend. The blue line of the proposed framework is always in the lowest range, and its stability advantage is more obvious. In the extreme perturbation scenario of uniform distribution, the processing time of some instances fluctuates by more than 40%. Min PDR and existing deep reinforcement learning scheduling methods show a significant spike in completion time, while the increase in completion time of the proposed framework is controlled within 10%, demonstrating strong robustness. This result is due to the design of the objective conditional reward function, which only depends on observable state transitions and not on pre-known processing times. This ensures that reward calculation remains effective in dynamic environments with arbitrary probability distributions, providing continuous and reliable learning feedback for the agent. The smooth trend of the visualization curve further confirms the adaptability of the proposed framework in complex dynamic scenarios.

[0096] The experiments also verified the effectiveness of the priority post-experience replay mechanism: HGNN+HER generates synthetic experience through target relabeling, which alleviates reward sparsity, but does not distinguish experience quality, resulting in a large number of suboptimal samples diluting the learning effect, and there are still significant fluctuations after policy convergence; HGNN+PER improves the utilization rate of high-value experience through priority sampling, but lacks effective utilization of failed experience, resulting in low learning efficiency in sparse reward scenarios; while the proposed framework combines the advantages of both, supplementing positive reward samples through HER and filtering high-learning-value experience through PER, making the policy convergence speed and scheduling optimization effect better than the variant model with a single experience replay mechanism. In the LA28 instance, the total completion time of the proposed framework is 1266, which is 8.8% lower than HGNN+HER (1389) and 9.7% lower than HGNN+PER (1402). Combined with the rapid convergence trend of with HER in the training performance curve, it fully demonstrates the rationality and superiority of the hybrid experience replay mechanism. It also shows that the combination of the HER mechanism and the priority mechanism achieves dual efficiency improvement in supplementing and selecting samples, further enhancing the learning efficiency and scheduling performance of the model.

[0097] Furthermore, ablation experiments further validated the core role of heterogeneous graph neural networks: after removing the heterogeneous graph encoder, the model used a feature matrix to represent the state, resulting in an average 12.5% ​​increase in total completion time across all instances, with more significant performance degradation in large-scale instances. This result, combined with the overall performance in both static and dynamic scenarios, demonstrates that heterogeneous graph modeling effectively integrates the heterogeneous features and topological relationships of processes and machines, avoiding the limitations of manual feature engineering. It provides more accurate state input for deep reinforcement learning and is a key foundation for the framework to maintain excellent performance in both static and dynamic scenarios. This also indirectly confirms its technological innovation and practicality in complex scheduling scenarios.

[0098] In summary, the workshop scheduling method based on heterogeneous graph neural networks and priority post-experience replay can significantly optimize the total completion time in static scenarios and exhibit excellent robustness and rescheduling capability in dynamic and uncertain environments. At the same time, the hybrid experience replay mechanism improves sample utilization efficiency and policy convergence speed, providing reliable technical support for flexible and intelligent production in modern manufacturing. All experimental results fully demonstrate the effectiveness, advancement, and scalability of the method.

[0099] The above are merely preferred embodiments of the present invention and are not intended to limit the scope of protection of the present invention. For those skilled in the art, the present invention can have various modifications and variations. Any changes, modifications, substitutions, integrations, and parameter changes made to these embodiments within the spirit and principles of the present invention, without departing from the principles and spirit of the present invention, through conventional substitutions or to achieve the same function, fall within the scope of protection of the present invention.

Claims

1. A workshop scheduling method based on heterogeneous graph neural networks and priority post-event experience replay, characterized in that, include: Load an instance of the dynamic job shop scheduling problem, the instance including a set of workpieces, a set of machines, a sequence of operations, a processing time matrix, and dynamic event settings; initialize the parameters of the deep reinforcement learning model, the model parameters including the weights of the heterogeneous graph neural network encoder, the Actor network, and the Critic network. In each training round, the scheduling environment is reset to its initial state, and at each decision time... The state coding, policy execution, environment interaction, and experience storage operations are executed sequentially. After the round ends, the post-experience replay engine is activated. For each original experience collected in the round, a target remarking strategy is used to generate synthetic experience, and the synthetic experience is stored in the HER experience buffer. A hybrid experience replay buffer is constructed, which is obtained by merging the original experience buffer and the HER experience buffer. A priority is calculated and maintained for each experience in the hybrid experience replay buffer, and the initial value of the priority is determined based on the absolute value of the time-series difference error. In each model update step, a small batch of experience samples is sampled from the hybrid experience replay buffer according to the priority and a preset probability. The total loss function of the PPO algorithm is calculated using the sampled experience batches, and the parameters of the Actor network and Critic network are optimized by gradient descent. Load the optimal model parameters after training, perform real-time scheduling decisions on new dynamic job shop scheduling instances, generate a complete scheduling scheme, and output the final completion time.

2. The workshop scheduling method based on heterogeneous graph neural networks and priority post-event experience replay as described in claim 1, characterized in that, The status coding includes: At the moment of decision The current scheduling environment state is constructed as a heterogeneous graph. , where the node set Includes process nodes and machine nodes edge set It includes three types of semantic edges: process sequence edges, compatibility edges, and occupancy edges. In dynamic job shop scheduling, the set of all processes that are to be processed or are being processed in the system at decision time $t$ is the process set. Configure time-varying attributes for the process node, including process ready status, remaining processing time, cumulative waiting time, and a binarized completion indicator; configure status attributes for the machine node, including machine availability status and remaining busy time. Heterogeneous graphs carrying node attributes The input is fed into a heterogeneous graph neural network encoder, which extracts an end-to-end feature representation of the current scheduling environment state through node embedding and information propagation in the graph neural network.

3. The workshop scheduling method based on heterogeneous graph neural networks and priority post-event experience replay as described in claim 2, characterized in that, The edge set The definitions and functions of the three types of semantic edges are as follows: The process sequence edge is a directed edge that uniquely connects adjacent process nodes within the same operation, and is used to explicitly depict the process sequence constraints of the operation. The compatibility edge is an undirected edge that connects each process node to all its selectable machine nodes, representing the potential processing location selection of the process. The compatibility edge is also configured with a binary feasibility indicator to indicate whether the corresponding machine can be assigned to the process. The occupied edge is an undirected edge that connects the machine node to the process node it is currently processing in real time, and is used to intuitively identify the real-time processing relationship between the machine and the process.

4. The workshop scheduling method based on heterogeneous graph neural networks and priority post-event experience replay as described in claim 2, characterized in that, The strategy execution includes: In the current scheduling state Below, based on heterogeneous graphs Dynamically construct a set of actionable actions based on node attributes and edge set constraints. The set of actionable actions consists of all valid process-machine assignment pairs, satisfying the constraint that ready processes are matched with idle and compatible machines. The heterogeneous graph The feature representation and the specified process completion target The inputs are fed into the Actor network, which calculates an unnormalized action score for each action in the set of possible actions. All action scores are then normalized using the softmax function to form an action probability distribution. Scheduling actions are obtained by random sampling based on the action probability distribution. The scheduling action is a unique process-machine allocation instruction.

5. The workshop scheduling method based on heterogeneous graph neural networks and priority post-event experience replay as described in claim 1, characterized in that, Each experience in the hybrid experience playback buffer priority and the preset sampling probability The calculation method is as follows: ,in For experience The timing difference error, A preset non-zero positive constant is used to ensure that all experiences have a non-zero sampling probability; ,in To control the hyperparameters of priority sampling intensity, The total number of experience samples currently stored in the mixed experience replay buffer. For the first The original priority of each empirical sample For the first The result of exponentially transforming the original priorities of a set of empirical samples.

6. The workshop scheduling method based on heterogeneous graph neural networks and priority post-event experience replay as described in claim 5, characterized in that, The timing difference error The calculation formula is: ,in For decision-making steps Instant rewards received This is a reward discount factor used to quantify the impact of future rewards on current decisions. This is the state-object pair value estimate output by the Critic network. For the parameters of the Critic network, To achieve the goal of completing the process based on original experience, To complete the target for the new process after the target has been remarked.

7. The workshop scheduling method based on heterogeneous graph neural networks and priority post-event experience replay as described in claim 6, characterized in that, The instant reward The target conditional reward function is calculated, and the specific calculation process of the target conditional reward function is as follows: Calculate the decision time Total waiting time for ready processes , ,in For the first A set of processes that are ready but not yet assigned. For process At any moment The cumulative waiting time; Then, calculate the immediate reward based on the completion status of the process. ,in A fixed positive reward upon completion of the process. The weighting coefficient for the waiting time penalty term. Let be the target indicator function, if The specified process is in status Completed in the middle If not completed, the value is 0.

8. The workshop scheduling method based on heterogeneous graph neural networks and priority post-event experience replay as described in claim 1, characterized in that, The total loss function of the PPO algorithm is: ,in Indicates time step Calculate the mathematical expectation from the empirical sample. and These are preset hyperparameters used to balance the weights of the value function loss and the entropy regularization term. For the network parameters to be optimized, For strategic losses, For the value function loss, This is the entropy regularization term.

9. The workshop scheduling method based on heterogeneous graph neural networks and priority post-event experience replay as described in claim 8, characterized in that, The strategy loss and value function loss The calculation formulas are as follows: ,in The ratio of the probabilities of the new and old strategies is given. For the new strategy in the state Select Action The probability, The probability of the old strategy. For the dominant function, For the cropping operation, The threshold value for the cropping operation; ,in For target value, For the valuation of the old Critic network, Hyperparameters for value tailoring.

10. The workshop scheduling method based on heterogeneous graph neural networks and priority post-event experience replay as described in claim 1, characterized in that, The real-time scheduling decision includes: At each decision point in the operation of the scheduling system, the current state of the scheduling environment is perceived and constructed into a heterogeneous graph according to the state coding method. The state feature representation is extracted by the heterogeneous graph neural network encoder; The state feature representation is input into the trained Actor network, and a deterministic strategy is used to select the action with the highest probability from the Actor network's output action probability distribution. The action Assign instructions to the optimal process-machine configuration in the current state; The action The command is sent to the actual production system or simulation environment for execution. The scheduling environment automatically evolves to the next state based on the execution result and real-time dynamic events. ; Repeat the above steps, making scheduling decisions based on the latest status at each new decision point, until all workpieces are processed or the scheduling period ends, generating a complete scheduling Gantt chart and process assignment list.

Images