Agv flexible workshop scheduling method and system based on layered heterogeneous graph reinforcement learning

Abstract

This invention discloses a method and system for AGV flexible workshop scheduling based on hierarchical heterogeneous graph reinforcement learning. The method includes: acquiring a set of processes, machines, and automated guided vehicles (AGVs) in a flexible workshop, and constructing a hierarchical heterogeneous graph scheduling model; introducing time-state dynamic updates to the hierarchical heterogeneous graph scheduling model, and embedding features of the process set, machine set, and AGV set to construct an embedded hierarchical heterogeneous graph scheduling model; generating scheduling decisions based on the embedded hierarchical heterogeneous graph scheduling model and executing them iteratively to generate a flexible workshop scheduling scheme. This invention improves the solution efficiency and stability of large-scale scheduling problems and achieves scalable intelligent scheduling decisions. As a method and system for AGV flexible workshop scheduling based on hierarchical heterogeneous graph reinforcement learning, this invention can be widely applied in the field of flexible workshop scheduling technology.

Description

Technical Field

[0001] This invention relates to the field of flexible workshop scheduling technology, and in particular to a flexible workshop scheduling method and system for AGVs based on hierarchical heterogeneous graph reinforcement learning. Background Technology

[0002] The Flexible Job Shop Scheduling Problem (FJSP) is a typical NP-hard complex problem in the field of manufacturing system scheduling. Its core lies in rationally allocating processing resources and determining the execution sequence of processes under the constraints of multiple workpieces, multiple processes, and multiple machines, in order to optimize overall production performance indicators, especially the maximum completion time (Makespan).

[0003] With the development of intelligent manufacturing and flexible logistics technologies, Automated Guided Vehicles (AGVs) are widely used for material transportation within workshops, extending the traditional Functional Junction Process (FJSP) problem into an FJSP-AGV problem that includes transportation resource constraints. In this problem, it is necessary not only to make decisions on the allocation of processing tasks between processes and machines, but also to simultaneously coordinate the transportation allocation and scheduling of AGVs, resulting in a high degree of coupling between processing and transportation decisions.

[0004] The FJSP-AGV problem can be formally described as follows: Given n workpieces, each workpiece contains several operations, m processing machines, and l AGVs, each operation can be processed on any machine in its candidate set. If adjacent operations are assigned to different machines, AGVs are needed for material transportation. The scheduling objective is to rationally allocate operations to machines, assign AGVs for transportation, and determine the processing order of each operation to minimize the maximum completion time.

[0005] The relevant technical solutions for FJSP-AGV mainly include the following categories: 1) Exact optimization methods: Mixed integer programming (MIP), constraint programming, and other methods are used to model and solve scheduling problems. These methods can obtain theoretically optimal solutions for small-scale problems, but their computational complexity increases exponentially at the actual workshop scale, making them difficult to apply to large-scale production environments.

[0006] 2) Heuristic and metaheuristic methods: such as genetic algorithms and particle swarm optimization, which search the feasible solution space by manually designing scheduling rules or operators. These methods rely on empirical rule design, have weak generalization ability, are difficult to adapt to dynamic state changes, and require parameter tuning for different problem instances.

[0007] 3) Deep Reinforcement Learning-Based Approaches: In recent years, some studies have combined Deep Reinforcement Learning (DRL) with Graph Neural Networks (GNNs) to model scheduling states using graph structures and learn end-to-end scheduling strategies. For example, the scheduling problem is modeled as a Markov decision process, using graph neural networks to extract state features; or a two-layer connection structure is used to reduce edge complexity; another example is applying GNNs and DRLs to dynamic scheduling problems with transportation resource constraints, employing a two-stream GNN architecture to fuse resource availability states.

[0008] However, most related technologies employ flat or fully connected graph structures, typically connecting processes, machines, and AGVs directly in pairs. This results in the number of graph edges increasing quadratically or even higher with the problem size, leading to excessively high computational complexity and difficulty in scaling to large-scale instances. Furthermore, existing methods generally fail to explicitly characterize the dynamic availability time state of machines and AGVs, resulting in a high proportion of invalid actions during scheduling decisions, and insufficient learning efficiency and scheduling stability. Summary of the Invention

[0009] To address the aforementioned technical problems, the present invention aims to provide a flexible workshop scheduling method and system for AGVs based on hierarchical heterogeneous graph reinforcement learning, thereby improving the efficiency and stability of solving large-scale scheduling problems and achieving scalable intelligent scheduling decisions.

[0010] The first technical solution adopted in this invention is: an AGV flexible workshop scheduling method based on hierarchical heterogeneous graph reinforcement learning, comprising the following steps: Obtain the process set, machine set, and automated guided vehicle set in the flexible operation workshop, and construct a hierarchical heterogeneous graph scheduling model; A time-state dynamic update hierarchical heterogeneous graph scheduling model is introduced, and the process set, machine set, and automated guided vehicle set are feature-embedded to construct the embedded hierarchical heterogeneous graph scheduling model. Based on the embedded hierarchical heterogeneous graph scheduling model, scheduling decisions are generated and executed cyclically to generate a flexible workshop scheduling scheme.

[0011] Furthermore, the step of obtaining the set of processes, machines, and automated guided vehicles in the flexible workshop and constructing a hierarchical heterogeneous graph scheduling model specifically includes: Obtain the set of processes, machines, and automated guided vehicles (AGVs) in the flexible work workshop, and determine the candidate machine set for each process, as well as the corresponding processing time and the AGV transportation time matrix between machines. Construct a set of process nodes to represent the processes to be processed; Construct a set of machine nodes to represent processing resources; Construct a set of automated guided vehicle (AGV) nodes to represent transportation resources; Connect the set of process nodes with the set of machine nodes to construct the process-machine edge set, which is used to represent that the process can be processed by the corresponding machine. Connect the set of machine nodes with the set of automated guided vehicles to construct a machine-automated vehicle edge set, which is used to represent that the machine can be provided with transportation services by the corresponding automated guided vehicle; A hierarchical heterogeneous graph scheduling model is constructed based on the set of process nodes, the set of machine nodes, the set of automated guided vehicle nodes, the set of process-machine edges, and the set of machine-automated guided vehicle edges.

[0012] Furthermore, the step of introducing a time-state dynamically updated hierarchical heterogeneous graph scheduling model and embedding features from the process set, machine set, and automated guided vehicle set to construct the embedded hierarchical heterogeneous graph scheduling model specifically includes: If a certain process is assigned to a machine and an automated guided vehicle in a flexible work workshop, the resource availability time of the hierarchical heterogeneous graph scheduling model is updated, and the connection relationship that does not meet the resource constraint is removed from the edge set to obtain the updated hierarchical heterogeneous graph scheduling model. For machine node embedding, the original features of the machine set are projected into the shared latent space through the multi-relational edge conditional attention mechanism, attention coefficients are calculated, and the attention weights are normalized before updating the machine set and embedding it into the updated hierarchical heterogeneous graph scheduling model. For the embedding of automated guided vehicle (AGV) nodes, the remaining available time of the AGV set is encoded into a temporal embedding vector, which is then element-wise added and layer-normalized and fused with the original nodes of the AGV set, and then embedded into the updated hierarchical heterogeneous graph scheduling model. For process node embedding, MLP is used to encode self-features, previous and subsequent process features, and candidate machine features. After concatenation, the final process is embedded into the updated hierarchical heterogeneous graph scheduling model, and the embedded hierarchical heterogeneous graph scheduling model is constructed.

[0013] Furthermore, the removal of connections from the edge set that do not satisfy resource constraints represents selecting an action at a certain scheduling moment, i.e., selecting... The selected processing steps are: The machine assigned to this process is The AGVs that perform transportation tasks are After updating the resource availability time of the hierarchical heterogeneous graph, the connections related to the allocated resources are removed from the edge set to reflect the new constraints after resource occupancy. The update rules are as follows: ; ; In the above formula, This indicates the index of the selected machine. This indicates the index of the selected AGV. The "difference" operator represents sets. Indicate process With machines One edge between them, Indicates machine With AGV One edge between them, Indicates at time Process node set With machine node set The set of edges between them Indicates at time Machine node set With AGV node set The set of edges between them Indicates at time After resource constraint pruning, the updated set of "process-machine" edges is obtained. Indicates at time After resource constraint pruning, the updated "machine-AGV" edge set is obtained.

[0014] Furthermore, the expression for calculating the attention coefficient is as follows: ; ; In the above formula, Indicates process node With machine nodes The attention coefficient (scalar) between the two measures the importance of their relationship. Represents machine node With AGV nodes Attention coefficient between them This represents a learnable attention weight vector. Indicates process node eigenvectors, Represents machine node eigenvectors, Indicates AGV node eigenvectors, Indicates the connection of process nodes and machine nodes The eigenvectors of the edges, Indicates the connection of machine nodes and AGV nodes The eigenvectors of the edges, This represents a nonlinear activation function used to introduce nonlinearity into the model.

[0015] Furthermore, the expression for updating the machine set after normalizing the attention weights is as follows: ; In the above formula, This indicates that after updating through a single layer of the graph attention network, the machine node... The new feature vector, Represents machine node The set of neighboring process nodes along the "process-machine" boundary. Indicates process node For machine nodes Normalized attention weights This represents the attention coefficient between processes and machines. Indicates AGV node , Refers to the serial number. Represents machine node The set of neighboring AGV nodes next to the "machine-AGV" node. Indicates AGV node For machine nodes Normalized attention weights This represents the attention coefficient between the machine and the AGV. Represents machine node Normalized attention weights for themselves Represents machine node The self-attention coefficient, This represents a nonlinear activation function used to perform a nonlinear transformation on the aggregated information to obtain the final node embedding.

[0016] Furthermore, the step of generating scheduling decisions and executing them cyclically based on the embedded hierarchical heterogeneous graph scheduling model to generate a flexible shop floor scheduling scheme specifically includes: Construct a scheduling state vector based on the embedded features in the embedded hierarchical heterogeneous graph scheduling model; The scheduling state vector is modeled as a probability distribution, and a priority score and constraint action masking mechanism are set. The probability of inactive actions is set to zero, and the loop is repeated until all processes are scheduled to be completed, thus generating a flexible workshop scheduling scheme.

[0017] Furthermore, the expression for modeling the scheduling state vector as a probability distribution is as follows: ; In the above formula, Indicating in strategy Under, state Choose action The probability, Represented by natural constant An exponential function with base 0. This represents a priority score function, implemented by a neural network (MLP), used to compute the priority score in the state. Below, each possible action Priority score, Indicates at time The scheduling action chosen by the agent. Indicates at time The state of the scheduling environment. This represents an action within the set of possible actions. Representing state The set of possible actions below Indicates action In state The priority score below, Indicates the state Below, one action from the set of possible actions. Priority score.

[0018] Furthermore, the specific expression of the reward function of the flexible workshop scheduling scheme is as follows: ; In the above formula, Describes an estimation function used to estimate the time required to schedule a given timeout. The maximum possible completion time (Makespan) for the entire scheduling scheme is as follows. This represents the reward function, used to quantify the state. Execute action Afterwards, transition to the new state. The immediate benefits gained by the intelligent agent during this process. Indicates at time The scheduling status contains complete information on all currently scheduled and unscheduled processes, machines, and AGVs. Indicates at time Selected scheduling action, Indicates at time Execute action Afterwards, at that moment The time scheduling system has entered a new state.

[0019] The second technical solution adopted in this invention is: an AGV flexible workshop scheduling system based on hierarchical heterogeneous graph reinforcement learning, comprising: The first module is used to obtain the set of processes, machines, and automated guided vehicles in the flexible work workshop and to build a hierarchical heterogeneous graph scheduling model. The second module is used to introduce a time-state dynamic update hierarchical heterogeneous graph scheduling model, and to embed features of the process set, machine set, and automated guided vehicle set to construct the embedded hierarchical heterogeneous graph scheduling model. The third module is used to generate scheduling decisions based on the embedded hierarchical heterogeneous graph scheduling model and execute them cyclically to generate a flexible workshop scheduling scheme.

[0020] The beneficial effects of the method and system of this invention are as follows: This invention constructs a hierarchical heterogeneous graph scheduling model by acquiring the process set, machine set, and automated guided vehicle set in a flexible workshop. The hierarchical heterogeneous graph structure significantly reduces the graph size and computational complexity. Furthermore, it introduces dynamic updates of the hierarchical heterogeneous graph scheduling model based on time states and embeds features of the process set, machine set, and automated guided vehicle set to construct an embedded hierarchical heterogeneous graph scheduling model. Explicitly introducing resource time states improves the accuracy of scheduling decisions and reduces ineffective scheduling actions through constraint guidance mechanisms, thereby improving learning efficiency. Finally, based on the embedded hierarchical heterogeneous graph scheduling model, scheduling decisions are generated and executed cyclically to generate a flexible workshop scheduling scheme, improving the solution efficiency and stability of large-scale scheduling problems and achieving scalable intelligent scheduling decisions. Attached Figure Description

[0021] Figure 1 This is a flowchart of the steps of the AGV flexible workshop scheduling method based on hierarchical heterogeneous graph reinforcement learning of the present invention; Figure 2 This is a structural block diagram of the AGV flexible workshop scheduling system based on hierarchical heterogeneous graph reinforcement learning of the present invention; Figure 3 This is a schematic diagram of the AGV flexible workshop scheduling process provided in a specific embodiment of the present invention; Figure 4 This is a schematic diagram of layered heterogeneity provided in a specific embodiment of the present invention. Detailed Implementation

[0022] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0023] First, it should be noted that the existing FJSP-AGV scheduling method has the following main shortcomings, for example: 1) Graph modeling has a complex structure and poor scalability: Existing methods generally adopt a fully connected or semi-fully connected structure of process-machine-AGV, which leads to a sharp increase in the number of edges in the graph with the number of nodes. In large-scale instances, the number of edges increases quadratically or even higher, resulting in excessively high computational costs and severely limiting the application of the methods in large-scale workshop scenarios.

[0024] 2) Insufficient characterization of resource time status: Most methods fail to explicitly model the dynamic availability time of machines and AGVs. The status is represented by a static graph structure, which cannot accurately reflect resource conflicts and waiting relationships, resulting in a lack of time awareness in scheduling decisions.

[0025] 3) High proportion of invalid scheduling actions: During reinforcement learning training, a large number of scheduling actions are infeasible due to violations of resource constraints (such as machines being occupied, AGVs being unavailable, etc.). Existing methods lack effective action pruning mechanisms, resulting in low exploration efficiency and slow training convergence. According to literature reports, the proportion of invalid actions can be as high as 40% or more.

[0026] 4) Scheduling decisions and resource coordination are not handled in a hierarchical manner: Existing solutions often treat process sequencing, machine allocation and AGV allocation as a single decision, resulting in a high decision space dimension, which increases the learning difficulty and computational complexity.

[0027] Based on this, such as Figure 3 As shown, this invention proposes an AGV-flexible job shop scheduling method (THG-RL) based on temporal hierarchical graph reinforcement learning. Through hierarchical heterogeneous graph modeling, time-aware feature embedding, and constraint-guided scheduling decision-making mechanism, it significantly reduces computational complexity while ensuring scheduling feasibility. It transforms the factorial complexity of the original problem into polynomial complexity, improves the solution efficiency and stability of large-scale scheduling problems, and realizes scalable intelligent scheduling decisions.

[0028] Reference Figure 1 This invention provides a flexible workshop scheduling method for AGVs based on hierarchical heterogeneous graph reinforcement learning, which includes the following steps: S100: Obtain the process set, machine set, and automated guided vehicle set in the flexible work workshop, and construct a hierarchical heterogeneous graph scheduling model; Specifically, the process set, machine set, and automated guided vehicle set in the flexible operation workshop are obtained, and the candidate machine set, corresponding processing time, and automated guided vehicle transportation time matrix between machines are determined. In this embodiment, the set of workpieces in the flexible workshop is obtained. Process set O, Machine set And Automated Guided Vehicle (AGV) collection Determine each process. candidate machine set and corresponding processing time And the AGV transport time matrix between machines. .

[0029] Furthermore, a set of process nodes is constructed to represent the processes to be processed; a set of machine nodes is constructed to represent processing resources; a set of automated guided vehicle (AGV) nodes is constructed to represent transportation resources; the set of process nodes and the set of machine nodes are connected to construct a set of process-machine edges to represent processes that can be processed by corresponding machines; the set of machine nodes and the set of AGVs are connected to construct a set of machine-AGV edges to represent machines that can be transported by corresponding AGVs; based on the set of process nodes, the set of machine nodes, the set of AGV nodes, the set of process-machine edges, and the set of machine-AGV edges, a hierarchical heterogeneous graph scheduling model is constructed.

[0030] In this embodiment, as Figure 4 As shown, a hierarchical heterogeneous graph containing the following nodes and edges is constructed. Specifically, it includes: 1) The set of process nodes O represents the process to be processed; 2) A set of machine nodes M, used to represent processing resources; 3) AGV node set A, used to represent transportation resources; 4) The process-machine edge set D represents the process that can be processed by the corresponding machine; 5) The machine-AGV edge set E represents the machine that can be transported by the corresponding AGV.

[0031] In this structure, process nodes are connected only to machine nodes, and AGV nodes are connected only to machine nodes, avoiding direct connections between processes and AGVs. This hierarchical structure reduces the number of edges from O(n×m×l) in a fully connected graph to O(n×m+m×l).

[0032] S200. Introduce a time-state dynamic update hierarchical heterogeneous graph scheduling model, and embed the features of the process set, machine set and automated guided vehicle set to construct the embedded hierarchical heterogeneous graph scheduling model. S210. If a certain process is assigned to a machine and an automated guided vehicle in a flexible work workshop, update the resource availability time of the hierarchical heterogeneous graph scheduling model and remove the connection relationships that do not meet the resource constraints from the edge set to obtain the updated hierarchical heterogeneous graph scheduling model. In this embodiment, during the scheduling process, the node time characteristics and feasible edge set are dynamically updated based on the events of process completion, machine occupancy, and AGV transportation completion. Specifically, when the process... Assigned to machine and AGV Then, update the available time of the resource: ; ; And remove connections from the edge set that do not satisfy resource constraints: ; ; In the above formula, This indicates the index of the selected machine. This indicates the index of the selected AGV. The "difference" operator represents sets. Indicate process With machines One edge between them, Indicates machine With AGV One edge between them, Indicates at time Process node set With machine node set The set of edges between them Indicates at time Machine node set With AGV node set The set of edges between them Indicates at time After resource constraint pruning, the updated set of "process-machine" edges is obtained. Indicates at time After resource constraint pruning, the updated "machine-AGV" edge set is obtained.

[0033] S220. For machine node embedding, the original features of the machine set are projected into the shared latent space through the multi-relational edge conditional attention mechanism, attention coefficients are calculated, and the attention weights are normalized before updating the machine set and embedding it into the updated hierarchical heterogeneous graph scheduling model. In this embodiment, machine node embedding employs a multi-relational edge conditional attention mechanism. First, the original features are projected into the shared latent space: ; ; ; The attention coefficient is calculated as follows: ; ; In the above formula, Indicates process node With machine nodes The attention coefficient (scalar) between the two measures the importance of their relationship. Represents machine node With AGV nodes Attention coefficient between them This represents a learnable attention weight vector. Indicates process node eigenvectors, Represents machine node eigenvectors, Indicates AGV node eigenvectors, Indicates the connection of process nodes and machine nodes The eigenvectors of the edges, Indicates the connection of machine nodes and AGV nodes The eigenvectors of the edges, This represents a nonlinear activation function used to introduce nonlinearity into the model.

[0034] Update machine node embeddings after normalizing attention weights: ; In the above formula, This indicates that after updating through a single layer of the graph attention network, the machine node... The new feature vector, Represents machine node The set of neighboring process nodes along the "process-machine" boundary. Indicates process node For machine nodes Normalized attention weights This represents the attention coefficient between processes and machines. Indicates AGV node , Refers to the serial number. Represents machine node The set of neighboring AGV nodes next to the "machine-AGV" node. Indicates AGV node For machine nodes Normalized attention weights This represents the attention coefficient between the machine and the AGV. Represents machine node Normalized attention weights for themselves Represents machine node The self-attention coefficient, This represents a nonlinear activation function used to perform a nonlinear transformation on the aggregated information to obtain the final node embedding.

[0035] S230. For the embedding of automated guided vehicle (AGV) nodes, the remaining available time of the AGV set is encoded into a temporal embedding vector and then fused with the original nodes of the AGV set element-wise and layer-normalized. The node embedding is then updated through a temporal attention mechanism and embedded into the updated hierarchical heterogeneous graph scheduling model.

[0036] In this embodiment, the AGV node timing embedding first includes the remaining available time of the AGV. Encoded as a temporal embedding vector: ; in, For learnable temporal coding weight matrix, For random noise that follows a Gaussian distribution, it is used for regularization.

[0037] The temporal embedding and the original node embedding are fused element-wise by addition and layer normalization: ; In the formula, AGV nodes after incorporating time-series information Embedded representation, AGV node The original embedding representation, Indicates AGV node The temporal embedding vector.

[0038] Furthermore, for AGV nodes after integrating time-series information... Calculate its temporal attention coefficient with that of its neighboring machine nodes: In the formula, AGV nodes after incorporating time-series information With machine nodes Attention coefficient between them This is a learnable attention weight vector. For machine nodes Embedded representation, To connect machine nodes and AGV nodes The edge embedding representation, This indicates vector concatenation.

[0039] The attention coefficient was normalized. ; In the formula, The attention weights are normalized. AGV node The set of neighboring machine nodes next to "machine-AGV".

[0040] Finally, the AGV node embedding is updated using a temporal attention mechanism: ; In the formula, For the updated AGV node The final embedding representation, It is a non-linear activation function. AGV node Learnable self-attention weights for oneself.

[0041] Through the aforementioned time-aware embedding mechanism, the embedded representation of AGV nodes can dynamically reflect their remaining available time status, thereby more accurately assessing the availability of AGVs and the urgency of transportation tasks in scheduling decisions.

[0042] In the above formula, This represents the result of element-wise addition of the original embedding and the temporal embedding. Presentation layer normalization operation, Indicates AGV node Temporal embedding vectors, Represents the learnable weight matrix of the temporal coding. Indicates AGV node The remaining available time, Indicates that it follows a Gaussian distribution random noise, This represents the AGV node after integrating timing information. The final embedding representation, Indicates AGV node The original embedding representation.

[0043] S240. For process node embedding, MLP is used to encode self-features, previous and subsequent process features and candidate machine features, and after splicing, the final process is embedded into the updated hierarchical heterogeneous graph scheduling model to construct the embedded hierarchical heterogeneous graph scheduling model.

[0044] In this embodiment, the process node embedding uses MLP-encoded self-features, preceding and following process features, and candidate machine features, which are then concatenated to generate the final process embedding.

[0045] S300 generates scheduling decisions based on the embedded hierarchical heterogeneous graph scheduling model and executes them cyclically to generate a flexible workshop scheduling scheme.

[0046] S310. Construct a scheduling state vector based on the embedded features in the embedded hierarchical heterogeneous graph scheduling model; In this embodiment, a scheduling state vector is constructed based on the above-mentioned embedded features: ; The scheduling strategy is modeled as a probability distribution, and the priority score is calculated by the MLP and transformed by softmax: ; In the above formula, Indicating in strategy Under, state Choose action The probability, Represented by natural constant An exponential function with base 0. This represents a priority score function, implemented by a neural network (MLP), used to compute the priority score in the state. Below, each possible action Priority score, Indicates at time The scheduling action chosen by the agent. Indicates at time The state of the scheduling environment. This represents an action within the set of possible actions. Representing state The set of possible actions below Indicates action In state The priority score below, Indicates the state Below, one action from the set of possible actions. Priority score.

[0047] By employing a constraint action masking mechanism, the probability of non-actions is set to zero, effectively reducing invalid exploration steps.

[0048] S320. Model the scheduling state vector as a probability distribution, set priority scores and constraint action masking mechanisms, set the probability of inactive actions to zero, repeat the loop until all processes are scheduled to be completed, and generate a flexible workshop scheduling scheme.

[0049] In this embodiment, the time state dynamic update step, feature embedding step, and scheduling decision generation step are repeatedly executed until all processes are scheduled and the final scheduling scheme is output. The reward function is defined as: ; in, This represents the maximum estimated completion time in the current state. Maximizing the cumulative reward is equivalent to minimizing the final completion time.

[0050] Therefore, the embodiments of the present invention can significantly reduce the graph size and computational complexity through a hierarchical heterogeneous graph structure; and explicitly introduce resource time states to improve the accuracy of scheduling decisions; reduce ineffective scheduling actions and improve learning efficiency through a constraint guidance mechanism; and can be applied to large-scale flexible workshop scheduling scenarios, with good scalability and engineering application value.

[0051] The embodiments of the present invention will be explained and illustrated with examples of engineering projects: This embodiment illustrates the implementation process of the method of the present invention using a simple FJSP-AGV example. Assume there are 3 workpieces in the workshop. , , 2 machines , 1 AGV.

[0052] workpiece It includes two processes. It includes two processes. It includes one process. The processing time is shown in the table below: ; Execution steps: First, construct a hierarchical heterogeneous graph, establishing connections between process nodes and their candidate machines, and between machine nodes and AGV nodes. Then, train the reinforcement learning policy network using the PPO algorithm. After training, apply the policy to new scheduling instances to generate scheduling schemes.

[0053] For large-scale applications, this embodiment demonstrates the application of this method on a large-scale scale. Consider a workshop with a scale of 30×15×3 (30 workpieces, 15 machines, and 3 AGVs). At this scale, the traditional fully connected graph method has O(30×15×3) = 1350 edges, while the method of this invention has O(30×15+15×3) = 495 edges, a reduction of approximately 63%.

[0054] Experimental results show that the completion time of this method on this large-scale instance is reduced by about 146% compared with traditional priority scheduling rules (such as FIFO, SPT, etc.), which is significantly better than existing methods.

[0055] For cross-scale generalization applications, this embodiment demonstrates the cross-scale generalization capability of the proposed method. A policy trained on 15×15×3 instances is directly applied to unseen instances of 40×15×3 scale, and good scheduling performance is still maintained, which proves the strong generalization capability of the proposed method.

[0056] In summary, the embodiments of the present invention have the following advantages over the prior art: 1) Significantly reduced computational complexity: By using a hierarchical heterogeneous graph structure, the edge complexity is reduced from O(n×m×l) of a fully connected graph to O(n×m+m×l), which can significantly reduce the graph size and improve computational efficiency in large-scale instances.

[0057] 2) Explicit time state modeling: The dynamic availability time of machines and AGVs is explicitly introduced through a time-aware embedding mechanism to improve the accuracy and timeliness of scheduling decisions.

[0058] 3) Significantly improved learning efficiency: By effectively pruning ineffective actions through the constraint action masking mechanism, invalid exploration is reduced, and training efficiency is improved by up to 58.9%.

[0059] 4) Good scalability and generalization ability: Experimental verification shows that the strategy trained on small-scale instances can be directly applied to large-scale unseen instances, and has good cross-scale generalization ability.

[0060] 5) Applicable to large-scale real-world scenarios: Experiments on benchmark datasets show that this method achieves a 6.55% to 8.93% improvement in completion time compared to traditional priority scheduling rules, with even more significant advantages in large-scale scenarios. Benchmark datasets vary in size, with an average size of (5×5×2). Using randomly generated large-scale instances (15×10×3 or more), the improvement compared to traditional priority scheduling rules (such as FIFO, SPT, MOR, MWKR) is over 40%. 6) High engineering application value: The method does not depend on the specific workshop layout, has strong versatility, and can be applied to various application scenarios such as offline scheduling and online rolling scheduling.

[0061] Reference Figure 2 A flexible workshop scheduling system for AGVs based on hierarchical heterogeneous graph reinforcement learning includes: The first module 201 is used to obtain the set of processes, the set of machines, and the set of automated guided vehicles in the flexible work workshop, and to construct a hierarchical heterogeneous graph scheduling model. The second module 202 is used to introduce a time-state dynamic update hierarchical heterogeneous graph scheduling model, and to embed the process set, machine set and automated guided vehicle set into features to construct the embedded hierarchical heterogeneous graph scheduling model. The third module 203 is used to generate scheduling decisions based on the embedded hierarchical heterogeneous graph scheduling model and execute them cyclically to generate a flexible workshop scheduling scheme.

[0062] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0063] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A flexible workshop scheduling method for AGVs based on hierarchical heterogeneous graph reinforcement learning, characterized in that, Includes the following steps: Obtain the process set, machine set, and automated guided vehicle set in the flexible operation workshop, and construct a hierarchical heterogeneous graph scheduling model; A time-state dynamic update hierarchical heterogeneous graph scheduling model is introduced, and the process set, machine set, and automated guided vehicle set are feature-embedded to construct the embedded hierarchical heterogeneous graph scheduling model. Based on the embedded hierarchical heterogeneous graph scheduling model, scheduling decisions are generated and executed cyclically to generate a flexible workshop scheduling scheme.

2. The AGV flexible workshop scheduling method based on hierarchical heterogeneous graph reinforcement learning according to claim 1, characterized in that, The step of obtaining the set of processes, machines, and automated guided vehicles in the flexible workshop and constructing a hierarchical heterogeneous graph scheduling model specifically includes: Obtain the set of processes, machines, and automated guided vehicles (AGVs) in the flexible work workshop, and determine the candidate machine set for each process, as well as the corresponding processing time and the AGV transportation time matrix between machines. Construct a set of process nodes to represent the processes to be processed; Construct a set of machine nodes to represent processing resources; Construct a set of automated guided vehicle (AGV) nodes to represent transportation resources; Connect the set of process nodes with the set of machine nodes to construct the process-machine edge set, which is used to represent that the process can be processed by the corresponding machine. Connect the set of machine nodes with the set of automated guided vehicles to construct a machine-automated vehicle edge set, which is used to represent that the machine can be provided with transportation services by the corresponding automated guided vehicle; A hierarchical heterogeneous graph scheduling model is constructed based on the set of process nodes, the set of machine nodes, the set of automated guided vehicle nodes, the set of process-machine edges, and the set of machine-automated guided vehicle edges.

3. The AGV flexible workshop scheduling method based on hierarchical heterogeneous graph reinforcement learning according to claim 2, characterized in that, The step of introducing a time-state dynamic update hierarchical heterogeneous graph scheduling model and embedding features from the process set, machine set, and automated guided vehicle set to construct the embedded hierarchical heterogeneous graph scheduling model specifically includes: If a certain process is assigned to a machine and an automated guided vehicle in a flexible work workshop, the resource availability time of the hierarchical heterogeneous graph scheduling model is updated, and the connection relationship that does not meet the resource constraint is removed from the edge set to obtain the updated hierarchical heterogeneous graph scheduling model. For machine node embedding, the original features of the machine set are projected into the shared latent space through the multi-relational edge conditional attention mechanism, attention coefficients are calculated, and the attention weights are normalized before updating the machine set and embedding it into the updated hierarchical heterogeneous graph scheduling model. For the embedding of automated guided vehicle (AGV) nodes, the remaining available time of the AGV set is encoded into a temporal embedding vector, which is then element-wise added and layer-normalized and fused with the original nodes of the AGV set, and then embedded into the updated hierarchical heterogeneous graph scheduling model. For process node embedding, MLP is used to encode self-features, previous and subsequent process features, and candidate machine features. After concatenation, the final process is embedded into the updated hierarchical heterogeneous graph scheduling model, and the embedded hierarchical heterogeneous graph scheduling model is constructed.

4. The AGV flexible workshop scheduling method based on hierarchical heterogeneous graph reinforcement learning according to claim 3, characterized in that, The removal of connections from the edge set that do not satisfy resource constraints represents selecting an action at a certain scheduling moment, i.e., choosing... The selected processing steps are: The machine assigned to this process is The AGVs that perform transportation tasks are After updating the resource availability time of the hierarchical heterogeneous graph, the connections related to the allocated resources are removed from the edge set to reflect the new constraints after resource occupancy. The update rules are as follows: In the above formula, This indicates the index of the selected machine. This indicates the index of the selected AGV. The "difference" operator for sets. Indicate process With machines One edge between them, Indicates machine With AGV One edge between them, Indicates at time Process node set With machine node set The set of edges between them Indicates at time Machine node set With AGV node set The set of edges between them Indicates at time After resource constraint pruning, the updated set of "process-machine" edges is obtained. Indicates at time After resource constraint pruning, the updated "machine-AGV" edge set is obtained.

5. The AGV flexible workshop scheduling method based on hierarchical heterogeneous graph reinforcement learning according to claim 4, characterized in that, The specific expression for calculating the attention coefficient is as follows: In the above formula, Indicates process node With machine nodes The attention coefficient (scalar) between the two measures the importance of their relationship. Represents machine node With AGV nodes Attention coefficient between them This represents a learnable attention weight vector. Indicates process node eigenvectors, Represents machine node eigenvectors, Indicates AGV node eigenvectors, Indicates the connection of process nodes and machine nodes The eigenvectors of the edges, Indicates the connection of machine nodes and AGV nodes The eigenvectors of the edges, This represents a nonlinear activation function used to introduce nonlinearity into the model.

6. The AGV flexible workshop scheduling method based on hierarchical heterogeneous graph reinforcement learning according to claim 5, characterized in that, The specific expression for updating the machine set after normalizing the attention weights is as follows: ； In the above formula, This indicates that after updating through a single layer of the graph attention network, the machine node... The new feature vector, Represents machine node The set of neighboring process nodes along the "process-machine" edge. Indicates process node For machine nodes Normalized attention weights This represents the attention coefficient between processes and machines. Indicates AGV node , Refers to the serial number. Represents machine node The set of neighboring AGV nodes next to "machine-AGV". Indicates AGV node For machine nodes Normalized attention weights This represents the attention coefficient between the machine and the AGV. Represents machine node Normalized attention weights for themselves Represents machine node The self-attention coefficient, This represents a nonlinear activation function used to perform a nonlinear transformation on the aggregated information to obtain the final node embedding.

7. The AGV flexible workshop scheduling method based on hierarchical heterogeneous graph reinforcement learning according to claim 6, characterized in that, The step of generating scheduling decisions and executing them cyclically based on the embedded hierarchical heterogeneous graph scheduling model to generate a flexible shop floor scheduling scheme specifically includes: Construct a scheduling state vector based on the embedded features in the embedded hierarchical heterogeneous graph scheduling model; The scheduling state vector is modeled as a probability distribution, and a priority score and constraint action masking mechanism are set. The probability of inactive actions is set to zero, and the loop is repeated until all processes are scheduled to be completed, thus generating a flexible workshop scheduling scheme.

8. The AGV flexible workshop scheduling method based on hierarchical heterogeneous graph reinforcement learning according to claim 7, characterized in that, The specific expression for modeling the scheduling state vector as a probability distribution is as follows: ； In the above formula, Indicating in strategy Under, state Choose action The probability, Represented by natural constant An exponential function with base 0. This represents a priority score function, implemented by a neural network (MLP), used to compute the priority score in the state. Below, each possible action Priority score, Indicates at time The scheduling action chosen by the agent. Indicates at time The state of the scheduling environment. This represents an action within the set of possible actions. Representing state The set of possible actions below Indicates action In state The priority score below, Indicates the state Below, one action from the set of possible actions. Priority score.

9. The AGV flexible workshop scheduling method based on hierarchical heterogeneous graph reinforcement learning according to claim 8, characterized in that, The specific expression for the reward function of the flexible workshop scheduling scheme is as follows: In the above formula, Describes an estimation function used to estimate the time required to schedule a given timeout. The maximum possible completion time (Makespan) for the entire scheduling scheme is as follows. This represents the reward function, used to quantify the state. Execute action Afterwards, transition to the new state. The immediate benefits gained by the intelligent agent during this process. Indicates at time The scheduling status contains complete information on all currently scheduled and unscheduled processes, machines, and AGVs. Indicates at time Selected scheduling action, Indicates at time Execute action Afterwards, at that moment The time scheduling system has entered a new state.

10. An AGV flexible workshop scheduling system based on hierarchical heterogeneous graph reinforcement learning, characterized in that, Includes the following modules: The first module is used to obtain the set of processes, machines, and automated guided vehicles in the flexible work workshop and to build a hierarchical heterogeneous graph scheduling model. The second module is used to introduce a time-state dynamic update hierarchical heterogeneous graph scheduling model, and to embed features of the process set, machine set, and automated guided vehicle set to construct the embedded hierarchical heterogeneous graph scheduling model. The third module is used to generate scheduling decisions based on the embedded hierarchical heterogeneous graph scheduling model and execute them cyclically to generate a flexible workshop scheduling scheme.

Images