Dynamic scheduling optimization method based on deep reinforcement learning and heterogeneous graph neural network

Abstract

The invention discloses a dynamic scheduling optimization method based on deep reinforcement learning and a heterogeneous graph neural network. The method comprises the following steps: firstly, converting a flexible job shop scheduling problem into a Markov decision process, and setting a state, an action, a state transition and a reward function; then constructing a basic heterogeneous graph and an enhanced heterogeneous graph, and defining the types and the number of nodes and edges and a directed relationship between the edges; performing three-stage feature embedding by adopting a heterogeneous graph attention network to obtain machine node embedding, operation node embedding, distribution instance node embedding and global state features; and inputting an action state vector formed after each feature is processed, optimizing a decision network according to a reward function, updating parameters to obtain a flexible job shop scheduling model, and completing FJSP solving. The method can effectively consider the flexible job shop scheduling problem under transport time and machine reachability constraints, can also process scheduling problems of different scales, has good generalization ability, and shows excellent performance in large-scale application.

Description

Technical Field

[0001] The present invention belongs to the field of industrial automation and intelligent manufacturing technology, and specifically relates to a flexible job shop dynamic scheduling optimization method and system based on deep reinforcement learning (DRL) and heterogeneous graph neural network (HGNN), which is particularly suitable for complex production environments under the constraints of transportation time and machine accessibility. Background Art

[0002] The Flexible Job Shop Scheduling Problem (FJSP) is a core challenge in industrial production. Its core task is to dynamically allocate compatible equipment to multiple operations and optimize the scheduling sequence to minimize the makespan. Traditional methods, such as heuristic rules (e.g., SPT and FIFO), lack global optimization capabilities. While mixed integer programming (MIP) can achieve exact solutions, its high computational complexity makes it difficult to cope with large-scale dynamic scenarios. Existing graph neural network-based methods often use static homogeneous graph modeling, which cannot effectively address transportation time, machine reachability constraints, and the cascading effects of dynamic scheduling, leading to scheduling conflicts and resource waste. Furthermore, their optimization objectives often only consider the impact of makespan, failing to coordinate makespan reduction, operation wait fairness, and equipment load balancing. Summary of the Invention

[0003] The purpose of this invention is to provide a dynamic scheduling optimization method based on deep reinforcement learning and heterogeneous graph neural networks, which realizes efficient scheduling under complex constraints through Markov decision process modeling, enhanced heterogeneous graph structure design and composite reward function.

[0004] The specific technical solutions of the present invention are as follows: Step (1) convert the flexible job shop scheduling problem into a Markov decision process, setting the state, action, state transition and reward function; Step (2) constructing a basic heterogeneous graph and an enhanced heterogeneous graph, defining the types, quantities, and connection relationships of nodes and edges; Step (3) uses a heterogeneous graph neural network to perform three-stage feature embedding to obtain machine node embedding features, operation node embedding features, allocation instance node embedding features and global state features; Step (4) constructs a decision network model, inputs the action state vector formed after processing the machine node embedding features, operation node embedding features, allocation instance node embedding features and global state features; the proximal policy optimization algorithm PPO optimizes the decision network according to the reward function, updates the parameters and obtains the flexible job shop scheduling model and completes the FJSP solution.

[0005] The setup of the Markov decision process described in step (1) is as follows: (1-1) state: at time step , environmental status It contains the following core information: the allocation status of all operations in the current scheduling plan; the remaining load and occupancy status of all machines; the set of all unscheduled operations and their readiness; and the deployment relationship between each operation and its compatible machines.

[0006] (1-2) Action: At each time step ,action Indicates the selection of a ready operation-processable equipment pair Make the allocation and start processing immediately, , Construct an action space for all candidate action sets, whose size gradually decreases during the scheduling process; Indicates the Homework No. operations, Indicates the A machine, , is the number of machines.

[0007] (1-3) State transition: Once the action is executed , the environment status updates the current related operations and machine information, and the environment status changes from Convert to ;

[0008] The basic heterogeneous graph described in step (2) Expressed as ; Represents the set of operation nodes, including all job operations and virtual start and end nodes (Start / End); Represents a set of machine nodes, where each physical device corresponds to an independent node; is an edge set, such as represents a directed temporal constraint edge set, which is used to describe the processing order of operations within a job, and It represents an undirected device association edge set, indicating the assignable relationship between operations and compatible machines.

[0009] Each environment state Represented as a basic heterogeneous graph and enhanced heterogeneous graphs ; where the node set , edge set 、 、 Changes dynamically as the solution process progresses.

[0010] The three-stage feature embedding described in step (3) is specifically as follows: Phase 1: For each machine node Generate machine node embedding features by aggregating the features of its adjacent operation nodes and the processing time features of the operation-machine edge through the improved Graph Attention Network v2 (GATv2); Phase 2: For each operation node The operation node embedding feature is generated by fusing its predecessor operation node feature, its own node feature, subsequent operation node feature and the embedding features of all its adjacent machine nodes through multi-channel multi-layer perceptrons (MLPs); Phase 3: For each allocated instance node In addition to its initial features, the corresponding machine embedding features are also spliced Embedded with operations Features are obtained, and allocation instance node embedding features are generated based on the aggregation of features of adjacent allocation instance nodes and transfer time features of allocation instance node-allocation instance node edges through GATv2.

[0011] The above-mentioned machine, operation, and allocation instance node embedding process is regarded as a layer of heterogeneous graph neural network (HGNN) operation. The structure is stacked with L layers to extract the features of the nodes under high-order neighbors layer by layer. After completing the L-layer feature update, the model average pools the final embeddings of all operation nodes and machine nodes respectively to obtain three fixed-dimensional vectors, which are then spliced ​​to generate fixed-dimensional global state features.

[0012] The decision network model in step (4) adopts the reinforcement learning structure based on actor-critic, which is composed of two multi-layer perceptrons (MLPs), which serve as the actor network and the critic network respectively. Their parameters are recorded as and , as follows: The actor network aims to generate a random policy , the calculation process of the entire strategy is divided into two steps: First, for each candidate action , integrating its related information into an action state vector input; the action state vector input includes operation Embedded features, machine Embedding features, distribution instances Embedded features and global state features ; Then, the concatenated action state vector is input into the actor network , calculate the scalar score corresponding to each action , used to guide action selection.

[0013] The critic network is used to estimate the current state The value of, the input is the global state feature , the output is the state value function , used for policy evaluation and gradient update.

[0014] The proximal policy optimization algorithm (PPO) evaluates the entire scheduling decision according to the reward function, updates the parameters of the actor network model and the critic network model, and obtains a flexible job shop scheduling model.

[0015] The present invention has the following beneficial effects and advantages: 1. Dynamic constraint adaptability: By enhancing the explicit modeling of transportation time and machine accessibility with heterogeneous graphs, scheduling delays caused by path selection or equipment conflicts can be effectively reduced. This approach is suitable for scenarios such as chemical and pharmaceutical industries that require multi-equipment collaboration and complex material transfer.

[0016] 2. Model generalization capability: Based on the improved graph attention network (GATv2) and dynamic pruning mechanism, it supports efficient scheduling of scalable production instances, avoiding the performance degradation of traditional methods caused by demand expansion, and is suitable for multi-batch, multi-product parallel manufacturing environments.

[0017] 3. Multi-objective collaborative optimization: A compound reward function coordinates the shortening of completion time, fairness in operation waiting, and equipment load balancing. This improves production efficiency while reducing the risk of system bottlenecks, adapting to industrial production needs that require a balance between efficiency and resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0019] Figure 1 A framework diagram of the method of the present invention; Figure 2 This is the isomeric graph that is the basis of the method of the present invention; Figure 3 Expand the heterogeneous graph for the method of the present invention (taking Task 1 as an example). DETAILED DESCRIPTION

[0020] The following is an explanation and description of the technical solutions of the embodiments of the present invention in conjunction with the drawings of the embodiments of the present invention. However, the following embodiments are only preferred embodiments of the present invention and are not exhaustive. Based on the embodiments in the implementation manner, other embodiments obtained by those skilled in the art without creative work are all within the scope of protection of the present invention.

[0021] Throughout this specification, the claims, and the accompanying drawings, the terms "first," "second," and so forth are used to distinguish between different items, not to describe a particular order. Furthermore, the term "comprises" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may include other steps or elements inherent to the process, method, product, or apparatus.

[0022] First, in this embodiment, the flexible job shop scheduling problem is targeted, and its instance consists of a set of jobs (Including jobs) and device collections (Including Each job Contains a set of operations that need to be performed according to a strict operation sequence Any operation Can be assigned to its compatible device set Any device in the process is processed, and the corresponding non-preemptive processing time is The scheduling goal is to allocate operations to equipment and determine their start time , so that the maximum completion time Minimize, where Indicates an operation The completion time of the machine is considered at the same time.

[0023] See Figure 1-3 , a dynamic scheduling optimization method based on deep reinforcement learning and heterogeneous graphs, comprising the following steps: The flexible job shop scheduling problem is converted into a Markov decision process, and the state, action, state transition, and reward function are set as follows: State: In the reinforcement learning framework, the representation of state is a key factor affecting the effectiveness of strategy learning. In order to accurately describe the Flexible Job Shop Scheduling (FJSP) at any time Based on the production environment information, this paper constructs a structured and dynamically changing state representation mechanism to support efficient scheduling decisions. , system status It contains the following core information: the allocation status of all operations in the current scheduling plan; the remaining load and occupancy status of all machines; the set of all unscheduled operations and their readiness; and the deployment relationship between each operation and its compatible machines.

[0024] In order to accurately reflect the progress of tasks in the scheduling process, a current partial scheduling solution is introduced. , used to dynamically track the status of each operation and estimate the start time of unscheduled operations: if the operation Allocation completed, starting time The actual processing time; if it is not scheduled, it is estimated based on its predecessor operation status and estimated processing time: ; This top-down recursive estimation can provide reasonable processing start time estimates for the operations that have not yet been assigned, thereby encoding their "expected execution order" in the state space. In addition, the current state also contains all candidate operation-device pairs. Scheduling feasibility information (i.e. and is currently idle), used to construct the action space.

[0025] Action: At each time step ,action Indicates the selection of a "ready operation-processable equipment pair" All candidate actions constitute the action space. , its size usually decreases gradually during the scheduling process.

[0026] Transition: Once the action is executed , about to operate Assign to device , the system status changes: Operation The status changes from "Ready" to "Processing"; the equipment Updated to "Working" status; if the next operation of the current job becomes schedulable, its status will also be updated to "Ready"; the overall status is transferred to This process is a deterministic transfer that continues until all operations are completed.

[0027] Reward function: To comprehensively consider target optimization, fairness, and system bottleneck risk control during the scheduling process, we introduce three reward indicators to construct a composite reward function: ; in, Represents the decrease in system makespan, used to maintain the original optimization goal; Represents the reduction of the average waiting time of each job, which encourages scheduling fairness and avoids operations that are not processed for a long time; Represents the change in device load imbalance (load variance), which measures whether the current scheduling has alleviated the system bottleneck; weight parameter Control the contribution strength of each sub-goal.

[0028] in, Indicates the The current load of each machine (allocated processing time + remaining processing task time).

[0029] in, For the The moment when the first schedulable operation of a job becomes ready, is the current time.

[0030] This reward function is designed to guide the agent to ensure the optimization of completion time while promoting fair task scheduling, reducing the risk of system bottlenecks, and making equipment utilization more balanced.

[0031] In order to better describe the scheduling status in the environment, a basic heterogeneous graph is used to display it. At the same time, an enhanced heterogeneous graph structure is proposed to address the constraints of transportation time and machine reachability.

[0032] The basic heterogeneous graph , Represents the set of operation nodes, including all job operations and virtual start and end nodes (Start / End); Represents a set of machine nodes, each physical device corresponds to an independent node, represents a directed temporal constraint edge set, which is used to describe the processing order of operations within a job, and It represents an undirected device association edge set, indicating the assignable relationship between operations and compatible machines. ,in Represents a set of allocated instance nodes, each node Indicates an operation In the machine An allocation instance on . Represents the transfer edge set between allocation instances, directed edges Represents an allocation instance arrive The allocation instance transfer relationship. To represent the instance binding edge, the ternary hyperedge Associated instance nodes , operation node and machine nodes ,make sure Unique Correspondence exist Execution instance on .

[0033] According to the above diagram and The definition of each state Represented as a heterogeneous graph and , where the node set , edge set 、 、 will change dynamically as the solution process progresses. Specifically, in Step selection action After that, only the representation operation remains With the selected machine The relationship between , while deleting the operation OM edges with all other machine nodes, and because the action Decisions have been clearly allocated so Except for the allocated instance node Delete the other related nodes. 、 There are also related deletions in . is a manifestation of the machine reachability constraint, which leads to Subsequent operations that are still unscheduled at the current time (such as 、 ...) of machines (e.g. 、 ...) and its OM side will adjust accordingly, which further leads to Assign instance nodes (such as 、 ...) and the update or removal of its corresponding edge structure. Thus, the updated graph is obtained and The state transition caused by action selection not only causes changes in the edges and assigned instance nodes corresponding to the current operation, but also affects the candidate selection of subsequent operations in a "cascade" manner through reachability constraints, resulting in the node adjacency structure in the graph changing dynamically over time.

[0034] At each time step , based on heterogeneous graph , define the operation The set of adjacent machines is ,machine The adjacency operation set is Based on heterogeneous graph , define the adjacent allocation instance node set of the allocation instance node as .

[0035] A three-stage feature embedding is performed using a heterogeneous graph neural network to obtain machine node embedding features, operation node embedding features, allocation instance node embedding features, and global state features, as follows: Machine node embedding: When constructing the embedding representation of a machine node, the important information carried by its adjacent operation nodes is first considered. For each machine node , whose adjacent nodes are a set of operations ,These operations have different effects on the current scheduling state, for example, those operations that are about to start have higher scheduling priority than those that are delayed.

[0036] To distinguish the importance of these operations, this paper introduces an improved Graph Attention Network v2 (GATv2) during the embedding process. This mechanism adaptively learns attention weights based on the feature differences of neighboring nodes. The original GATv2 architecture only considers node features and does not involve edge information. Therefore, this paper customizes and expands it to simultaneously process node features and the processing time characteristics of operation-machine edges.

[0037] Specifically, for an operation With machines The connecting edge , whose features are operated by the original features The processing time characteristics of the edge Splicing composition Machine nodes and operation nodes use different linear transformations and , and then calculate the importance score of the adjacent operation to the machine through nonlinear activation and attention parameters .

[0038] .

[0039] Furthermore, the influence of the machine itself on its representation is also preserved and Indicates that the attention weights are finally obtained through softmax normalization 、 , used to aggregate the representation information of adjacent operation nodes to generate updated machine embedding .

[0040] ; .

[0041] Operation Node Embedding: Operation Node The adjacency structure of is more complex, including both previous and next operations in the same job (connected by directed edges) and the compatible machine nodes connected to it (connected by undirected edges). Due to the different sources of information, directly using the graph attention mechanism is not applicable.

[0042] To this end, a multi-channel multi-layer perceptron (MLP) architecture was designed to process and fuse information from different neighbors separately. Five MLPs are used to process: ① the features of the predecessor operation; ② the original features of the operation node itself; ③ the features of the subsequent operation; and the embedded features of all adjacent machine nodes (first summed and then used as input). The final features are concatenated and projected into a fixed-dimensional operation embedding.

[0043] This approach effectively integrates multi-source heterogeneous features while preserving the node structure information, making the final operation embedding More expressive.

[0044] ; in, 、 、 、 、 .

[0045] Assignment instance node embedding: For the assignment instance node, its initial features cannot well show the global information, so this paper designs that in addition to the initial features, the features in the embedding process should also be spliced ​​with machine embedding Embedded with operations ,Right now At the same time, for a certain allocation instance Assign instance with next operation The connecting edge , whose features are composed of the concatenated features The transfer time characteristics of the edge Splicing composition, that is Using linear transformation , and then calculate the importance score of the adjacent assignment instance to the node through nonlinear activation and attention parameters .

[0046] .

[0047] Furthermore, the influence of the machine itself on its representation is also preserved and Indicates that the attention weights are finally obtained through softmax normalization 、 , used to aggregate the representation information of adjacent allocation instance nodes and generate updated allocation instance node embeddings .

[0048] ; .

[0049] Stacking and pooling: The above machine, operation and allocation instance node embedding process can be regarded as a layer of heterogeneous graph neural network (HGNN) operation. To further improve the representation ability of the model, this paper stacks the structure layers (parameters of each layer are not shared with each other), thereby extracting the features of nodes under high-order neighbors layer by layer.

[0050] In the entire embedding process, the initial features of the operation node and the machine node are used in the first layer input, and the initial features of the instance node and the edge features are assigned. 、 It is used as the input for attention calculation in all layers.

[0051] Finish After the layer features are updated, the model performs mean pooling on the final embeddings of all operation nodes, machine nodes, and allocation instance nodes to obtain three fixed-dimensional vectors, which are then concatenated into a global representation of the heterogeneous graph state. , serves as the input of the policy network for subsequent scheduling decisions.

[0052] .

[0053] On the other hand, the critic network is used to estimate the current state The value of . Its input is the global state feature , the output is the state value function , used for policy evaluation and gradient update.

[0054] The actor-critic network model is optimized according to the reward function using the PPO algorithm, and the environment state, network parameters, and heterogeneous graph are updated according to the selected action.

[0055] The training process is shown in Algorithm 1. In each iteration, multiple FJSP instances are randomly sampled. The agent extracts state features using a heterogeneous graph neural network and selects action-machine pairs for scheduling via a policy network. After scheduling, the generalized advantage estimation (GAE) is used to calculate the advantage of each step, and the policy and value network parameters are updated based on the PPO loss function. During training, policy validation and data resampling are performed regularly to improve generalization performance and stability. The parameters used are shown in Table 1.

[0056] Table 1. Hyperparameter configuration of deep reinforcement learning model

[0057] Finally, the optimized flexible workshop scheduling model is obtained. When solving the actual FJSP, the original parameters of each dimension are input into the model to obtain an optimal path corresponding to the minimum value of the maximum completion time, that is, the best solution of the required FJSP is obtained, which guides the production efficiency to the highest level.

[0058]

[0059] To further verify the comparability and effectiveness of the proposed scheduling strategy, a mixed integer programming (MIP) model for the flexible job shop scheduling problem was constructed as a benchmark for the exact solution. This model aims to minimize the system's maximum makespan and considers the following variables and constraints: Table 2: MIP model elements

[0060] Objective function: .

[0061] Main constraints: Each operation must be assigned to exactly one compatible machine: ; Path accept constraint (next operation selects a machine): ; Path sending constraints (current operation transfer decision): ; Time relationship constraints: ; ; ; Inter-operation timing constraints: ; Machine conflict constraints: (different jobs on the same machine do not overlap): ; ; ; Maximum completion time constraint: .

[0062] To validate the effectiveness of this embodiment, we first used training at different scales to obtain different solution models, including 10×5, 15×10, 20×5, and 20×10. To demonstrate the generalization performance and solution quality of the present invention, we evaluated the trained policy on instances of the same size as the training set, as well as on instances larger than the training set but not seen during training, including 30×10 and 40×10. We then compared our approach with four state-of-the-art scheduling rules, using the minimum completion time after solving the MIP model as a benchmark.

[0063] Table 3: Solution results

[0064] As can be seen from the above table, the algorithm proposed in the present invention has a higher solution quality than the scheduling rules in most cases. Therefore, the algorithm proposed in the present invention for solving the flexible job shop scheduling problem has greatly improved the generalization ability and guaranteed the solution quality.

Claims

1. A dynamic scheduling optimization method based on deep reinforcement learning and heterogeneous graph neural networks, characterized by: Step (1) convert the flexible job shop scheduling problem into a Markov decision process, setting the state, action, state transition and reward function; Step (2) constructing a basic heterogeneous graph and an enhanced heterogeneous graph, defining the types, quantities, and connection relationships of nodes and edges; Step (3) uses a heterogeneous graph neural network to perform three-stage feature embedding to obtain machine node embedding features, operation node embedding features, allocation instance node embedding features and global state features; Step (4) constructs a decision network model, inputs the action state vector formed after processing the machine node embedding features, operation node embedding features, allocation instance node embedding features and global state features; the proximal policy optimization algorithm PPO optimizes the decision network according to the reward function, updates the parameters and obtains the flexible job shop scheduling model, and completes the FJSP solution.

2. The dynamic scheduling optimization method based on deep reinforcement learning and heterogeneous graph neural network according to claim 1, characterized in that: The setup of the Markov decision process described in step (1) is as follows: (1-1) state: at time step , environmental status Contains the following core information: the allocation of all operations in the current scheduling plan; the remaining load and occupancy status of all machines; the set of all unscheduled operations and their readiness; the deployment relationship between each operation and its compatible machines; (1-2) Action: At each time step ,action Indicates the selection of a ready operation-processable equipment pair Make the allocation and start processing immediately, , Construct an action space for all candidate action sets; Indicates the Homework No. operations, Indicates the A machine, , is the number of machines; (1-3) State transition: Once the action is executed , the environment status updates the current related operations and machine information, and the environment status changes from Convert to ; (1-4) Reward function: Construct a composite reward function; where represents the decrease in system completion time, Indicates that at time step When the environmental state Estimated completion time for the following projects: Indicates the reduction value of the average waiting time of each job, Indicates that at time step When the environmental state The average waiting time of all jobs under Indicates the change in the device load imbalance. Indicates that at time step When the environmental state The load imbalance of the machine; weight parameter Control the contribution strength of each sub-goal.

3. The dynamic scheduling optimization method based on deep reinforcement learning and heterogeneous graph neural network according to claim 2, characterized in that: The basic heterogeneous graph described in step (2) Expressed as ; Represents the set of operation nodes, including all job operations and virtual start and end nodes; Represents a set of machine nodes, where each physical device corresponds to an independent node; is an edge set, such as represents a directed temporal constraint edge set, which is used to describe the processing order of operations within a job, and It represents an undirected device association edge set, indicating the assignable relationship between operations and compatible machines; The enhanced heterogeneous graph Expressed as ; Represents a set of allocated instance nodes, each of which Indicates an operation In the machine An allocation instance on ; Represents the transfer edge set between allocation instances, directed edges Represents an allocation instance To the allocation instance transfer relationship, Indicates instance binding edge, ternary hyperedge Associated instance nodes , operation node and machine nodes ,make sure Unique Correspondence exist Execution instance on ; Each environment state Represented as a basic heterogeneous graph and enhanced heterogeneous graphs ; where the node set , edge set 、 、 Changes dynamically as the solution process progresses.

4. The dynamic scheduling optimization method based on deep reinforcement learning and heterogeneous graph neural network according to claim 3 is characterized in that: The three-stage feature embedding described in step (3) is specifically as follows: Phase 1: For each machine node Generate machine node embedding features by aggregating the features of its adjacent operation nodes and the processing time features of the operation-machine edge through the improved graph attention network GATv2; Phase 2: For each operation node The operation node embedding feature is generated by fusing its predecessor operation node feature, its own node feature, subsequent operation node feature and the embedding features of all adjacent machine nodes through multi-channel multi-layer perceptrons (MLPs); Phase 3: For each allocated instance node In addition to its initial features, the corresponding machine embedding features are also spliced Embedded with operations Features are obtained, and allocation instance node embedding features are generated based on the aggregation of features of adjacent allocation instance nodes and transfer time features of allocation instance node-allocation instance node edges through GATv2.

5. The dynamic scheduling optimization method based on deep reinforcement learning and heterogeneous graph neural network according to claim 4 is characterized in that: The decision network model in step (4) adopts an actor-critic reinforcement learning structure, which consists of two multi-layer perceptrons (MLPs), which serve as the actor network and the critic network respectively. Their parameters are recorded as and , as follows: The actor network aims to generate a random policy , the calculation process of the entire strategy is divided into two steps: First, for each candidate action , integrating its related information into an action state vector input; the action state vector input includes operation Embedded features, machine Embedding features, distribution instances Embedded features and global state features ; Then, the concatenated action state vector is input into the actor network , calculate the scalar score corresponding to each action , used to guide action selection; The critic network is used to estimate the current state The value of, the input is the global state feature , the output is the state value function , used for strategy evaluation and gradient update; The proximal policy optimization algorithm (PPO) evaluates the entire scheduling decision according to the reward function, updates the parameters of the actor network model and the critic network model, and obtains a flexible job shop scheduling model.

6. The dynamic scheduling optimization method based on deep reinforcement learning and heterogeneous graph neural network according to claim 4, characterized in that: The embedding process of machine, operation and assignment instance nodes in step (4) is regarded as a layer of heterogeneous graph neural network operation. The structure is stacked with L layers to extract the features of nodes under high-order neighbors layer by layer; After completing the L-layer feature update, the model performs average pooling on the final embeddings of all operation nodes and machine nodes to obtain three fixed-dimensional vectors, which are then concatenated to generate fixed-dimensional global state features.

Images