Dynamic distributed flexible workshop scheduling method based on hierarchical multi-agent deep reinforcement learning

Abstract

The present invention discloses a dynamic distributed flexible job shop scheduling method based on hierarchical multi-agent deep reinforcement learning, belonging to the field of distributed job shop scheduling technology. The method comprises: determining the initial state of the dynamic distributed flexible job shop scheduling problem to be solved; when a new workpiece arrives, constructing a global heterogeneous graph to represent the relationship between the new workpiece and the FMU; the upper-level agent uses a hierarchical graph attention architecture to extract the features of the new workpiece and FMU through a process-level attention module, a workpiece-level attention module, and FMU encoding, and inputs the features into a decision network to obtain the probability of the FMU action, and selects the FMU with the highest probability; assigning the new workpiece to a designated FMU and updating the original features of the FMU as well as the machine and process features within the FMU; constructing an underlying heterogeneous graph, and using a heterogeneous graph neural network to predict feasible actions and select an appropriate process and machine combination. The present invention can efficiently solve the distributed job shop scheduling problem of continuous arrival of dynamic workpieces, collaborative scheduling of multiple flexible manufacturing units, and real-time response.

Description

Technical Field

[0001] The present invention relates to the field of distributed workshop scheduling technology and also to industrial application software, specifically a dynamic distributed flexible workshop scheduling method based on hierarchical multi-agent deep reinforcement learning. Background Art

[0002] With the rapid development of intelligent manufacturing, the manufacturing industry is undergoing a transition from large-scale standardized production to large-scale personalized customization. In this process, the flexibility and responsiveness of manufacturing systems have become crucial. In the context of a globalized economy, supply chain networks are becoming increasingly complex, and cross-regional collaboration is becoming more frequent. This requires manufacturing systems to not only maintain efficient operation at the local level but also collaborate in a distributed environment. Therefore, solving the dynamic distributed flexible job-shop scheduling problem (2DFJSP) has become a key issue in optimizing modern manufacturing systems. Specifically, 2DFJSP addresses the collaborative work between multiple factories or flexible manufacturing units (FMUs). These FMUs are geographically distributed and each possesses a variety of machine types capable of performing similar or different processing operations. When a new task arrives, the system must determine which FMU will accept the task and determine the specific processing path, namely, which machine to select to complete each process step. This process not only considers the current status of all machines but also requires real-time decision-making based on dynamically changing manufacturing information to avoid unnecessary waiting time and resource waste.

[0003] In current production scheduling, the most common techniques for addressing distributed flexible job shop scheduling problems are priority dispatching rules (PDR) and metaheuristic methods. PDR relies on simple mathematical calculations and is suitable for real-time decision-making, but it lacks flexibility and is not suitable for complex, large-scale scheduling environments. Metaheuristic methods, on the other hand, typically operate as centralized scheduling strategies and produce high-quality solutions, but typically require significant computational time and are unsuitable for dynamic, real-time scheduling.

[0004] With the development of deep reinforcement learning (DRL), a growing number of researchers are experimenting with combining DRL with graph neural networks to solve job shop scheduling problems, achieving promising results. However, existing research still has some limitations. Existing methods often rely on local information or fixed rules, ignoring the complex interactions between workpieces, machines, and the production environment. They lack effective global feature extraction frameworks, fail to fully exploit the potential of real-time data and intelligent decision-making, and suffer from poor response speed and flexibility when processing real-time workpieces. Summary of the Invention

[0005] In response to the low adaptability and efficiency of existing technologies for solving the 2DFJSP problem, the present invention introduces an end-to-end hierarchical multi-agent deep reinforcement learning framework (Hierarchical Multi-Agent Deep Reinforcement Learning Framework, HMADRL) to provide a dynamic distributed flexible workshop scheduling method based on hierarchical multi-agent deep reinforcement learning. The method can extract key information from the multi-dimensional information of machines, unscheduled processes and newly arrived workpieces, and designs a hierarchical decision-making mechanism in combination with the rescheduling mechanism to enhance the decision-making ability and rapid response capability of the manufacturing system in complex dynamic environments, thereby achieving more efficient, flexible and adaptable real-time scheduling.

[0006] In order to achieve the above object, the solution provided by the present invention is as follows:

[0007] A dynamic distributed flexible workshop scheduling method based on hierarchical multi-agent deep reinforcement learning includes the following steps:

[0008] S1. For the dynamic distributed flexible shop scheduling problem to be solved, determine the initial state of the manufacturing system, including the characteristics of all machines, unassigned processes and FMUs;

[0009] S2. When a new workpiece arrives, a global heterogeneous graph is constructed to represent the relationship between the new workpiece and each FMU based on the processing system information at the current decision moment. The upper-level intelligent agent observes the state and selects actions according to the Markov decision process, and uses a hierarchical graph attention architecture to extract the information of the processing system at the current moment. The hierarchical graph attention architecture extracts the features of the new workpiece and FMU through the process-level attention module, the workpiece-level attention module and the FMU encoding, and obtains the probability of the new workpiece being assigned to each FMU through the decision network, and then assigns the new workpiece to the appropriate FMU.

[0010] S3. After assigning the new workpiece to the specified FMU, schedule the unassigned processes in the FMU together with the processes of the new workpiece, and update the original features of the FMU as well as the machine and process features in the FMU;

[0011] S4. For each FMU, a low-level heterogeneous graph is constructed to represent the relationship between the machines and processes in each FMU at the current moment. The low-level agent observes the state and selects actions based on the Markov decision process. The low-level agent makes decisions using a heterogeneous graph neural network and selects the appropriate process and machine combination by predicting feasible actions. The state is represented by the low-level heterogeneous graph, including the characteristics of the processes and machines in the FMU.

[0012] As a specific embodiment of the present invention, a new workpiece includes at least one process. The global heterogeneous graph accurately reflects the processing characteristics and compatibility differences of the workpiece on different FMUs by modeling the relationship between each process and different FMUs as a meta-path. The original features of each unassigned process, machine, and FMU are defined as follows:

[0013] The original features of the unassigned process include: processing status, the number of adjacent machines in each FMU, the average processing time in each FMU, the number of processes corresponding to the workpiece of the process, the estimated completion time of the workpiece corresponding to the process in each FMU, and the estimated start time of the workpiece corresponding to the process in each FMU;

[0014] The original characteristics of each machine in each FMU include: the number of unprocessed adjacent processes, the average processing time of the remaining adjacent processes, and the standard deviation of the processing time of the remaining adjacent processes;

[0015] The original features of each FMU include: processing progress, process completion rate, average workpiece completion rate, standard deviation of workpiece completion rate, average utilization rate of machines in the FMU, and standard deviation of machine utilization rate in the FMU;

[0016] Furthermore, the upper-level intelligent agent is composed of a hierarchical graph attention architecture, which effectively captures the complex interactive relationship between processes, workpieces and FMUs by introducing hierarchical attention mechanisms at the process level and workpiece level, while making dynamic scheduling decisions on a global scale.

[0017] Furthermore, the process-level attention is to capture the importance of the compatible machines in each FMU to the process. First, the importance of the machine to the process and the importance of the process to itself are calculated, and then the embedding of the process node is calculated. Considering the meta-path set {φ1, Φ2,…, φ l ,…,φ f}, after executing the process-level attention h times, the final process node embedding is obtained

[0018]

[0019] e ij =LeakyReLu(aT(Wo μ ij ∥W o μ ij ))

[0020]

[0021] in, is the machine k in the flexible manufacturing cell l for process O ij The attention coefficient; LeakyReLu (·) is the activation function; a, W o , W m is a learnable parameter; T represents transposition; ∥ represents connection; μ ij For process O ij The original characteristics of is the original feature of machine k in flexible manufacturing cell l, OM arc Connected node pairs The original characteristics of e ij For process O ij Attention coefficient to oneself; For process O ij Embedding of process nodes of flexible manufacturing unit l; σ(·) is the activation function sigmod; is the machine k in the flexible manufacturing cell l; For operation O ij Available machines in the neighborhood; O ij is the j-th process of workpiece i; To use the softmax function Parameters after normalization; α ij To use the softmax function for e ij Parameters after normalization;

[0022] Artifact-level attention captures the importance of the same artifact to different FMUs and aggregates the feature embeddings of the same artifact to different FMUs to determine the final artifact embedding;

[0023] First, the importance of the meta-path between each process and the FMU is calculated

[0024]

[0025] Where W represents the weight matrix; b represents the bias vector; q is the artifact-level attention vector; It is process O ij Embedding of flexible manufacturing unit l; Process O ijThe importance of meta-path to flexible manufacturing cell l; yes Normalized by the softmax function; tanh(·) is the activation function; is the embedding of workpiece i into the meta-path of flexible manufacturing unit l, n i is the total number of processes for job i, Z i is the embedding of artifact i;

[0026] To encode the FMU, three multi-layer perceptrons (MLPs) with the same structure and ELU activation are used to process the feature vector of the FMU node and the embedded feature vector of the new workpiece, thereby obtaining the embedded feature vector Z of the FMU node. l , then, calculate the graph pooling vector Z t .

[0027]

[0028] Among them, Z l is the embedding of flexible manufacturing unit l; It is used to process Z i Multilayer Perceptron; It is used to process Z l Multi-layer perceptron; MLP ψ is a multilayer perceptron that processes the embedded vector after ELU activation; ELU(·) is the activation function; is the embedding of workpiece i after stacking L layers of graph attention network layers; It is U after stacking L layers of graph attention network layers l Embedding of U l It is a flexible manufacturing unit l; It is a collection of flexible manufacturing units; is the number of flexible manufacturing units; Z t is the graph pooling vector.

[0029] The upper agent policy network will Z t As input, to select the appropriate FMU action, the policy network consists of a multi-layer perceptron MLP and produces action selection probabilities

[0030]

[0031] As a specific embodiment of the present invention, the reward function in the upper-layer agent is as follows:

[0032]

[0033] in, is the reward value, is the maximum load at a certain state, the state is in brackets, s t 、s t+1 are the states corresponding to time steps t and t+1 respectively; Indicates the assigned process O ij In the machine The actual processing time, represents the set of processes that have been allocated in the flexible manufacturing unit 1, represents the set of unassigned processes in flexible manufacturing unit 1, Indicates any unassigned process O ij in U l Processing time in.

[0034] Furthermore, the underlying heterogeneous graph is constructed to represent the relationship between machines and processes in the current FMU, where the original features of the process nodes are related to the global heterogeneous Figure 1 The original characteristics of a machine consist of three items: the number of adjacent processes, the completion time of the machine before the current time t, and the proportion of the machine usage time relative to the current time t.

[0035] Furthermore, the underlying agent is composed of a heterogeneous graph neural network, which takes the machine node embedding and process node embedding as the policy network input, outputs actions, and selects the specified OM pair (process-machine pair);

[0036] Furthermore, the machine node aggregates the feature vectors of adjacent process nodes and edges to generate the final embedding vector of the machine. MLP is applied to encode the original feature vector of each process node to obtain the process node feature embedding. After calculating the embedding of the process node and the machine node, the graph pooling vector is calculated. The calculation formula of the graph pooling vector is as follows:

[0037]

[0038] Where h t is the graph pooling vector; O new The process for the newly arrived workpiece; is the process set of the newly arrived workpiece; It is the set of processes that are not assigned to the current FMU; Embedding process node features; is the total number of machines; For machines The final embedding vector of .

[0039] Furthermore, the underlying agent outputs all feasible OM pair probabilities through the policy network, adopts sampling or greedy decoding strategy, and selects actions according to the obtained distribution. The calculation formula of OM pair probability is as follows:

[0040]

[0041] in, is the probability of action selection of the lower-level agent, and MLP(·) is a multi-layer perceptron.

[0042] Furthermore, the reward function in the underlying agent is as follows:

[0043]

[0044] Where, Indicates the maximum completion time of flexible manufacturing unit l at a certain moment; LB * (O ij ) represents the process O at a certain moment ij The scheduled completion time or the minimum estimated completion time of the unscheduled operations.

[0045] Furthermore, both layers of agents are trained using the Proximal Policy Optimization (PPO) algorithm; a staged training procedure is adopted: the bottom-level agent is first trained on fixed-size instances, and the upper-level agent is subsequently trained using the policy derived from the bottom-level agent, focusing on global artifact distribution.

[0046] Beneficial effects:

[0047] This method describes the potential load status of machines and FMUs using a heterogeneous graph customized for newly arrived jobs. This accurately captures the processing characteristics and compatibility differences of each workpiece across different FMUs. This description approach not only considers the current FMU status and production schedule, but also fully accounts for the resource requirements of the workpiece, providing comprehensive decision support for the 2DFJSP problem.

[0048] The upper-level agent of our invention is composed of a hierarchical graph attention architecture that extracts critical information from multiple dimensions, including information about machines, unscheduled processes, and newly arrived workpieces. By employing dual attention modules at both the process and workpiece levels, this framework effectively captures the complex relationships between new work and FMUs. This allows the system to quickly and accurately allocate workpieces in the face of dynamic events, achieving global awareness of the production process and precise control of local decision-making.

[0049] This paper develops an end-to-end hierarchical multi-agent deep reinforcement learning framework. The HMADRL framework decomposes the 2DFJSP problem into multiple static instances of the flexible job shop scheduling problem, with lower-level agents handling each of these sub-problems in their own FMU. This approach allows the system to gradually solve complex scheduling tasks while maintaining global control of the entire system, ensuring consistent and coherent decision-making. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 A flowchart of a dynamic distributed flexible workshop scheduling method based on hierarchical multi-agent deep reinforcement learning provided by the present invention;

[0051] Figure 2 This is a heterogeneous graph in which workpiece nodes are embedded in an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and embodiments.

[0053] like Figure 1 As shown, the present invention provides a dynamic distributed flexible workshop scheduling method based on hierarchical multi-agent deep reinforcement learning, including the following contents:

[0054] A dynamic distributed flexible workshop scheduling method based on hierarchical multi-agent deep reinforcement learning includes the following steps:

[0055] S1. For the dynamic distributed flexible shop scheduling problem to be solved, determine the initial state of the manufacturing system, including the characteristics of all machines, unassigned processes and FMUs;

[0056] S2. When a new workpiece arrives, a global heterogeneous graph is constructed to represent the relationship between the new workpiece and each FMU based on the processing system information at the current decision moment. The upper-level intelligent agent observes the state and selects actions according to the Markov decision process, and uses the layered graph attention architecture to extract the information of the processing system at the current moment. This architecture extracts the features of the new workpiece and FMU through the process-level attention module, the workpiece-level attention module and the FMU encoding, and obtains the probability of the new workpiece being assigned to each FMU through the decision network, and then assigns the new workpiece to the appropriate FMU.

[0057] S3. After assigning the new workpiece to the specified FMU, schedule the unassigned processes in the FMU together with the processes of the new workpiece, and update the original features of the FMU as well as the machine and process features in the FMU;

[0058] S4. For each FMU, a low-level heterogeneous graph is constructed to represent the relationship between the machines and processes in each FMU at the current moment. The low-level agent observes the state and selects actions based on the Markov decision process. The low-level agent makes decisions using a heterogeneous graph neural network and selects the appropriate process and machine combination by predicting feasible actions. The state is represented by the low-level heterogeneous graph, including the characteristics of the processes and machines in the FMU.

[0059] Each step will be further explained below.

[0060] First, the 2DFJSP problem is expressed as follows: a series of workpieces J = {J1, J2, ..., J i …,J n}Continuous arrival, J represents the workpiece set, J * The subscript represents the workpiece, and the subscript is the serial number of the workpiece. These workpieces must be in the flexible manufacturing unit (FMU) U = {U1, U2, ..., U l ,…,U f} system, U represents the flexible manufacturing unit set, U * Represents a flexible manufacturing unit, and the subscript is the serial number of the flexible manufacturing unit. Each FMU can represent a factory in a geographically dispersed manufacturing network or a production line in a flexible manufacturing unit. Each FMU contains a group of machines M* represents the collection of machines in a flexible manufacturing unit, and the superscript is the serial number of a flexible manufacturing unit. Represents the machine, the superscript is the serial number of the flexible manufacturing unit, and the subscript is the serial number of the machine. Each workpiece i contains n i Continuous process O i ={O i1 ,…,O ij ,…,O ini}, with precedence constraints, O i is the set of processes for workpiece i, O i* The first number in the subscript is the serial number of the workpiece, and the second number is the serial number of the process of the workpiece. Process O can be executed on the same subset of computers that meet the conditions in various FMUs. ij Process O ij The processing time p ijk (denoting the processing time of the jth operation of the i-th job on machine k) depends on the specific FMUl and machine k. When a job arrives, its earliest available time is determined, and all its operations must be assigned to the same FMU. In this case, 2DFJSP aims to assign incoming jobs to FMUs, determine the order of operations in each FMU, and select a machine for each operation to minimize the total completion time C max , defined as the maximum completion time among all factories, i.e. ,in Indicates that in the flexible manufacturing unit U l Process O ij Completion time, n i is the total number of processes for workpiece i, n is the total number of workpieces, and f is the total number of flexible manufacturing units.

[0061] 1. Constructing a global heterogeneous graph of 2DFJSP

[0062] First, a global heterogeneous graph needs to be constructed to represent the relationship between the new workpiece and the flexible manufacturing unit (FMU). The global heterogeneous graph is represented as like Figure 2 (Heterogeneous graph of artifact node embedding) as shown, is the set of unassigned processes for newly arrived workpieces, It is the collection of all FMU machines. and machine nodes They each have original eigenvectors and represents a six-dimensional vector, Represents a three-dimensional vector. In addition, a working node is introduced and FMU node sets To represent the new work and the comprehensive characteristics of each FMU. It should be noted that only the flexible manufacturing unit node Provides the original feature vector of the FMU The embedding of new work is formed by the aggregation of machine subnodes and operation subnodes. is a set of connected arcs, ε is composed of arcs with original features The compatible machine arc group is composed of process O ij in U l The processing time on the k-th machine in . OM arc Connected node pairs The original characteristics of Indicates the connection process O ij With machines arc.

[0063] 2. Constructing the upper Markov process of the upper agent

[0064] At each time step t(t∈{1,…,n}), the upper agent observes the system state and select an action It can assign workpieces to a selected FMU, and the bottom-level agent then schedules all unassigned processes in the selected FMU together. Through this mechanism, the 2DFJSP is decomposed into multiple static FJSPs. The Markov decision model of the upper-level agent:

[0065] (a) Status At a given time step t, the state Represented as a heterogeneous graph The graph includes the original features of all processes, machines and FMUs. In order to characterize the importance of different machines to the process, it is necessary to combine the features of the process nodes with the machine features within each FMU. l Unassigned process node O ij Defines a unique primitive feature Process O ij Get different state representations in different FMUs. exist The original eigenvectors in To reflect The potential processing load at time step t. In addition, capturing the total potential workload of each FMU is crucial for the upper-level agent, so we define U for each rescheduling point t l The original eigenvector of The primitive features of each unassigned operation, machine, and FMU are defined as follows:

[0066] Unassigned process O ij U l The original characteristics

[0067] Processing state: binary value, at time t if O ij Already scheduled for processing, it is represented as 1, otherwise it is 0.

[0068] in U l The number of adjacent machines in:

[0069] in U l Average processing time in: in It is process O ij in U l The set of available machines in For process O ij Processing time on machine k of flexible manufacturing cell 1.

[0070] Workpiece J i Number of steps in: n i .

[0071] Workpiece J i in U l Estimated completion time in: where at i Workpiece J i Arrival time.

[0072] Estimated start time:

[0073] machine The original characteristics

[0074] Number of adjacent operations that have not been processed: in It's U l The number of adjacent steps remaining in , Is the workpiece that is not assigned and can be used in the machine The number of processing steps.

[0075] machine Average processing time of the remaining adjacent operations It is the processing time of the workpiece i to be processed on the machine k of the l-th flexible manufacturing unit.

[0076] machine Standard deviation of the processing time of the remaining adjacent operations

[0077]

[0078] U l The original characteristics

[0079] U l Processing progress in: in It's U l The total processing time of the completed operations, is the average processing time of the remaining operations.

[0080] U l Completion rate of intermediate process: where n l (t) is U l The number of jobs at the rescheduling point t, OP i (t) is workpiece J i The number of operations currently completed.

[0081] U l Average completion rate of workpieces: CRJ i (t) is the job J at time step t when rescheduling i The completion rate is calculated as follows: CRJ i (t) = OP i (t) / n i .

[0082] Standard deviation of workpiece completion rate:

[0083]

[0084] U l Average utilization of machines in: in At time step t, U l The utilization of machine k in A feature of .

[0085] U l The standard deviation of machine utilization in:

[0086]

[0087] (b) Action: Whenever a new workpiece arrives at time step t, the upper agent selects an action Assign the workpiece to the appropriate FMU.

[0088] (c) Reward: Our goal is to learn how to allocate new workpieces to reduce the global completion time. We define the reward as and The maximum load of all FMUs The difference between them, that is: in is defined as follows:

[0089]

[0090] in Indicates the assigned process O ij In the machine The actual processing time, Indicates U l The set of processes that have been assigned in Indicates U l The set of unassigned steps in Indicates any unassigned process O ij in U l The processing time in the schedule is used, while the actual processing time of the scheduled operation is used. The total reward is calculated by summing up the instantaneous rewards across all time steps. This method encourages the upper-level agent to assign workpieces to relatively idle FMUs or FMUs with shorter processing times, thereby reducing the total completion time span.

[0091] (d) State transfer: According to and The environment of the FMU that triggers the rescheduling will move deterministically towards the next state. In the subsequent decision step t+1, the upper-level agent verifies whether there is a new workpiece arriving at the current system time of each FMU. When there is at least one new workpiece in the queue of all FMUs, the state is recorded. And select actions based on the designed layered graph attention architecture Newly arrived workpieces are immediately assigned to the designated FMU.

[0092] 3. Applying Hierarchical Graph Attention Architecture Decision

[0093] This framework adopts a hierarchical graph attention structure, from process-level attention to workpiece-level attention, and effectively captures the complex interactive relationship between workpieces and production resources through hierarchical information processing and attention mechanism.

[0094] (a) Process-level attention module: The process-level attention module captures the compatible machines in each FMU for process O. ij The importance of the sub-node representation is integrated, and for each process node, f process node embeddings corresponding to each FMU are generated. It is worth noting that the representation of each operation node is different between FMUs. Then, the model uses f embeddings to identify the different FMU meta-paths for process O. ij The relative importance of the processes is determined and their features are extracted to produce the final process feature embedding.

[0095] Given by OM arc Connected node pairs we will The original characteristics and The original characteristics Connect to form Each adjacent machine For process O ij The importance is measured as follows:

[0096]

[0097] in, and are learnable parameters, T Indicates transposition, ∥ indicates connection. ij The neighborhood of O must also include itself. ij Importance to itself is calculated as follows:

[0098] e ij =LeakyReLu(a T (W o μ ij ∥W o μ ij ))

[0099] Use the softmax function to and e ij Normalization is performed to obtain the weight coefficient parameter and αij Then, the embedding of the process node is calculated by summing the mapped features of the adjacent nodes and scaling them by their respective coefficients. As shown below:

[0100]

[0101] Consider the set of meta-paths {φ1,φ2,…,φ l ,…,φ f}, after executing the process-level attention h times, the final embedding is obtained

[0102] (b) Artifact-level attention module: heterogeneous graph Each process node in carries semantic information across multiple FMUs. In order to emphasize the semantic information of multiple FMUs and a given job J i The interaction between i Process embedding for each FMU Aggregation is first performed to produce FMU-specific artifact node embeddings Each process and U l The importance of the meta-path between is quantified by:

[0103]

[0104] Where W is the weight matrix, b is the bias vector, and q is the artifact-level attention vector. l The weight of the meta-path is expressed as It is connected to the U l The importance of all meta-paths These weighted coefficients are then used to combine the FMU-specific process embeddings:

[0105]

[0106] The workpiece embedding of FMU is obtained by calculation Workpiece J i Each process and U l The importance of the meta-paths between can be averaged to represent U l To J i overall importance.

[0107] Finally, the feature embeddings of different FMUs for the same workpiece are aggregated to determine the final workpiece embedding Z i :

[0108]

[0109] (c) FMU encoding: In heterogeneous graphs In the embedded U l The eigenvector ξ l Therefore, three multi-layer perceptrons (MLPs) with the same structure and ELU activation are used to process the FMU node U l The feature vector of the new workpiece and the embedded feature vector of the FMU node U are obtained. l The embedded feature vector Z l :

[0110]

[0111] in It is used to process Z i Multilayer perceptron, It is used to process Z l Multilayer Perceptron, MLP ψ It is a multi-layer perceptron that processes the embedded vector after ELU activation, and ELU(·) is the activation function.

[0112] To enhance the feature extraction capability, L layers of hierarchical graph attention network layers with the same structure are stacked together to embed FMU nodes and unassigned workpiece nodes. Subsequently, the graph pooling vector Z is calculated. t as follows:

[0113]

[0114] in, is the embedding of workpiece i after stacking L layers of graph attention network layers; It is U after stacking L layers of graph attention network layers l Embedding is the number of flexible manufacturing units.

[0115] (d) Decision: The policy network takes the embedding vector as input to select the appropriate FMU action The policy network consists of MLP layers and generates action selection probabilities

[0116]

[0117] The obtained probabilities are then normalized using the softmax function. During policy training, a random sampling strategy is used to explore various actions, while greedy decoding is used during policy evaluation.

[0118] 4. Markov Decision Process of the Underlying Agent

[0119] (a) State: The state at time step t Heterogeneous Graph Indicates that Assigned to U l Process node O ij The original characteristics Inherited If O ij Already arranged in M k Processing, then its processing time will become the actual value In addition, J i The feature of the number of operations in will be updated to the count of its unscheduled operations. i Already completed, is the actual value of its completion time, and Indicates that process O has been scheduled ij The actual start time of processing.

[0120] machine The original characteristics It consists of three items: means The number of adjacent processes, means The completion time before t, means The ratio of usage time to the current time t.

[0121] (b) Action: Action Represents the composite decision at time step t Assign a matching operation to the machine Processing. The action space depends on the previous process O i(j-1) The number of eligible processes that have been completed, and the number of idle machines at time t.

[0122] (c) Reward: To minimize the maximum completion time of the static subproblem, the reward function is designed as the difference between the completion times of the partial solutions at time steps t and t+1 (the current scheduling situation in the processing system at that moment, that is, the scheduling results of the assigned processes, excluding the unassigned processes), that is, in represents time t U l The maximum completion time, LB t (O ij ) represents process O ij Scheduled completion time or the minimum estimated completion time of unscheduled operations

[0123] (d) State transition: After taking work, the environment transitions to the next state During this conversion process, if there is no feasible OM pair at the current system time, the system time will advance to the next earliest available machine time. Otherwise, continue to select from the feasible OM pairs. At step t+1, the graph structure only retains the process node O ij of arc, while removing other compatible pairs.

[0124] 5. Applying Heterogeneous Graph Neural Network Decision Making

[0125] (a) Machine node embedding: In the machine The neighbors are a set of processes Their processing time is longer than other feature pairs Since the processing time information represented by the OM arc is crucial, we incorporate edge attention into This method preserves the feature information of adjacent edges based on the relationship strength of adjacent edges. In addition, heterogeneous node attention is used to learn the importance of different process nodes. Combining these two aggregation modes, we can obtain a method that preserves the feature information of adjacent process nodes and edges. Embedding vectors, details are as follows.

[0126] ① Heterogeneous edge attention: In order to obtain edge We aggregate the context information of its connected nodes O ij 、 and the edge The eigenvectors are calculated Feature Embedding

[0127]

[0128] in, and Is the mapping transformation parameter weight matrix between process nodes and machine nodes. Using edge feature embedding As input, the side attention coefficient can be calculated as follows. It is obtained by embedding the edge feature into the vector input:

[0129]

[0130] Next, for all Are all normalized by the softmax function, generating coefficients Then, the aggregate edge feature vector for each machine is:

[0131]

[0132] ② Heterogeneous node attention: For each machine The calculation formula of node attention coefficient is:

[0133]

[0134] machine Its own attention coefficient is the same as O ij The importance of itself is calculated. Then the normalized attention coefficient is obtained through the softmax function. and α kk These coefficients are used to summarize the characteristics of neighboring nodes:

[0135]

[0136] Finally, the feature vectors of adjacent process nodes and edges are aggregated to generate the machine The final embedding vector of is expressed as:

[0137]

[0138] (b) Process node embedding: The machine node embedding contains the feature information of the process node. Therefore, we apply MLP to encode the original feature vector of each process node to obtain the process node feature embedding After computing the embeddings of the process nodes and machine nodes, the graph pooling vector is calculated as follows:

[0139]

[0140] where h t is the graph pooling vector; O new The process for the newly arrived workpiece; is the process set of the newly arrived workpiece; It is the set of processes that are not assigned to the current FMU; Embedding process node features; is the total number of machines; For machines The final embedding vector of

[0141] (c) Action prediction: To determine feasible actions We introduce a policy network to map the scheduling state to a probability distribution and derive the probability of all feasible OM pairs. The calculation formula is as follows:

[0142]

[0143] in is the probability of action selection of the lower-layer agent, and MLP(·) is a multi-layer perceptron. Finally, a sampling or greedy decoding strategy is used to select an action based on the obtained distribution.

[0144] 6. Training of upper-level and lower-level agents

[0145] Both layers of agents are trained using PPO. The PPO algorithm is a reinforcement learning method based on the Actor-Critic framework. The Actor is trained based on the current state s. t and probability to choose the appropriate action a t , interact with the environment, and obtain the next state s t+1 and reward r t ; Critic calculates the value function of the current state and the value function of the next state Indicates that at a specific π θ Next, from the state At the beginning, the expected return that the algorithm can obtain. The upper-level agent optimizes the objective function L(θ) as follows:

[0146]

[0147] Among them, r t (θ) is the probability ratio, is the advantage function, To take the expectation, clip(·) is the clipping function, ∈ is the hyperparameter for adjusting the clipping interval, θ is the policy network parameter, γ t′ is the discount factor at time t', is the reward of the upper agent at time t', is the state of the upper agent at time t, is the action of the upper agent at time t, T is the total time step, π θ (·) is the new strategy, For the old strategy, v φ (·) is the output function of the upper-layer agent critic network.

[0148] A staged training procedure is adopted: the bottom-level agent is first trained on fixed-size instances. The upper-level agent is then trained using the policy derived from the bottom-level agent, focusing on global artifact distribution.

[0149] In this embodiment, the manufacturing information shown in Table 1 is used to train HMADRL, where the job arrival is represented by a Poisson process, so that the time interval between arriving workpieces follows an exponential distribution. The average arrival time interval is specified as where μ a represents the average processing time of all processes in the workpiece, μ ois the average number of steps per workpiece, is the load factor of the job shop.

[0150] Table 1 Example manufacturing information

[0151]

[0152] The performance of the trained HMADRL is evaluated by comparing it with three commonly used scheduling methods (PDR, metaheuristic algorithm and DRL method).

[0153] Table 2 lists the average target value, gap percentage, and running time of all methods on 20 instances at each scale, where the instance size is expressed as The gap percentage is calculated relative to the minimum target value among all methods.

[0154] As shown in the table, the proposed method consistently outperforms all baseline methods under the six test scales.

[0155]

[0156] The above are only preferred specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed in the embodiments of the present invention should be covered within the scope of protection of the present invention.

Claims

1. A dynamic distributed flexible workshop scheduling method based on hierarchical multi-agent deep reinforcement learning, characterized by: The steps include: S1. For the dynamic distributed flexible shop scheduling problem to be solved, determine the initial state of the manufacturing system, including the characteristics of all machines, unassigned processes and FMUs; S2. When a new workpiece arrives, a global heterogeneous graph is constructed to represent the relationship between the new workpiece and each FMU based on the machining system information at the current decision moment. The upper-level agent observes the state and selects actions according to the Markov decision process, using a hierarchical graph attention architecture to extract information about the machining system at the current moment. This architecture extracts the features of the new workpiece and FMU through process-level attention modules, workpiece-level attention modules, and FMU encoding. The decision network then determines the probability of the new workpiece being assigned to each FMU and assigns the new workpiece to the appropriate FMU. S3. After assigning the new workpiece to the specified FMU, schedule the unassigned processes in the FMU together with the processes of the new workpiece, and update the original features of the FMU as well as the machine and process features in the FMU; S4. Construct an underlying heterogeneous graph for each FMU to represent the relationship between the machines and processes in each FMU at the current moment. The underlying intelligent agent observes the state and selects actions based on the Markov decision process, uses the heterogeneous graph neural network to make decisions, and selects the appropriate process and machine combination by predicting feasible actions. The state is represented by the underlying heterogeneous graph, including the characteristics of the processes and machines in the FMU.

2. A dynamic distributed flexible workshop scheduling method based on hierarchical multi-agent deep reinforcement learning according to claim 1, characterized in that: The new workpiece includes at least one process. The global heterogeneous graph models the relationship between each process and different FMUs as a meta-path. The original features of each unassigned process, machine, and FMU are defined as follows: The original features of the unassigned process include: processing status, the number of adjacent machines in each FMU, the average processing time in each FMU, the number of processes corresponding to the workpiece of the process, the estimated completion time of the workpiece corresponding to the process in each FMU, and the estimated start time of the workpiece corresponding to the process in each FMU; The original characteristics of each machine in each FMU include: the number of unprocessed adjacent processes, the average processing time of the remaining adjacent processes, and the standard deviation of the processing time of the remaining adjacent processes; The original features of each FMU include: processing progress, process completion rate, average workpiece completion rate, standard deviation of workpiece completion rate, average utilization rate of machines in the FMU, and standard deviation of machine utilization rate in the FMU.

3. The dynamic distributed flexible workshop scheduling method based on hierarchical multi-agent deep reinforcement learning according to claim 1 is characterized in that: The hierarchical graph attention architecture of the upper-layer agent effectively captures the complex interactions between processes, workpieces, and FMUs by introducing hierarchical attention mechanisms at the process and workpiece levels, while making dynamic scheduling decisions at a global scale. The process-level attention is to capture the importance of the compatible machines in each FMU to the process. First, the importance of the machine to the process and the importance of the process to itself are calculated, and then the embedding of the process node is calculated. Considering the meta-path set {φ l }, l∈(1,f), after executing the process-level attention h times, the final process node embedding is obtained Where, f is the total number of flexible manufacturing units, is the machine k in the flexible manufacturing cell l for process O ij The attention coefficient; LeakyReLu (·) is the activation function; a, W o , W m is a learnable parameter; represents transposition; || represents connection; μ ij For process O ij The original characteristics of is the original feature of machine k in flexible manufacturing cell l, OM arc Connected node pairs The original characteristics of e ij For process O ij Attention coefficient to oneself; For process O ij Embedding of process nodes of flexible manufacturing unit l; σ(·) is the activation function sigmod; is the machine k in the flexible manufacturing cell l; For operation O ij Available machines in the neighborhood; O ij is the j-th process of workpiece i; To use the softmax function Parameters after normalization; α ij To use the softmax function for e ij Parameters after normalization; Artifact-level attention captures the importance of the same artifact to different FMUs and aggregates the feature embeddings of the same artifact to different FMUs to determine the final artifact embedding; First, the importance of the meta-path between each process and the FMU is calculated Where W represents the weight matrix; b represents the bias vector; q is the artifact-level attention vector; It is process O ij Embedding of flexible manufacturing unit l; Process O ij The importance of meta-path to flexible manufacturing cell l; yes Normalized by the softmax function; tanh(·) is the activation function; is the embedding of workpiece i into the meta-path of flexible manufacturing unit l, n i is the total number of processes for job i, Z i is the embedding of artifact i; To encode the FMU, three multi-layer perceptrons (MLPs) with the same structure and ELU activation are used to process the feature vector of the FMU node and the embedded feature vector of the new workpiece, thereby obtaining the embedded feature vector Z of the FMU node. l , then, calculate the graph pooling vector Z t； Among them, Z l is the embedding of flexible manufacturing unit l; Is used to process Z i Multilayer Perceptron; It is used to process Z l Multi-layer perceptron; MLP ψ is a multilayer perceptron that processes the embedded vector after ELU activation; ELU(·) is the activation function; is the embedding of workpiece i after stacking L layers of graph attention network layers; It is U after stacking L layers of graph attention network layers l Embedding of U l It is a flexible manufacturing unit l; It is a collection of flexible manufacturing units; is the number of flexible manufacturing units; Z t is the graph pooling vector; The upper agent policy network will Z t As input, to select the appropriate FMU action, the policy network consists of a multi-layer perceptron MLP and produces action selection probabilities 4. A dynamic distributed flexible workshop scheduling method based on hierarchical multi-agent deep reinforcement learning according to claim 3, characterized in that: The reward function in the upper agent is as follows: in, is the reward value, is the maximum load at a certain state, the state is in brackets, s t 、s t+1 are the states corresponding to time steps t and t+1 respectively; Indicates the assigned process O ij In the machine The actual processing time, represents the set of processes that have been allocated in the flexible manufacturing unit 1, represents the set of unassigned processes in flexible manufacturing unit 1, Indicates any unassigned process O ij Processing time in flexible manufacturing cell 1.

5. The method for dynamic distributed flexible workshop scheduling based on hierarchical multi-agent deep reinforcement learning according to claim 2 is characterized in that: The underlying heterogeneous graph is constructed to represent the relationship between machines and processes in the current FMU. The original features of the process nodes are consistent with the global heterogeneous graph. The original features of the machine consist of three items: the number of adjacent processes, the completion time of the machine before the current time t, and the ratio of the machine usage time to the current time t.

6. The method for dynamic distributed flexible workshop scheduling based on hierarchical multi-agent deep reinforcement learning according to claim 5, characterized in that: The underlying intelligent agent is composed of a heterogeneous graph neural network, which takes machine node embedding and process node embedding as policy network input, outputs actions, and selects specified OM pairs.

7. The method for dynamic distributed flexible workshop scheduling based on hierarchical multi-agent deep reinforcement learning according to claim 6 is characterized in that: The machine node aggregates the feature vectors of adjacent process nodes and edges to generate the final embedding vector of the machine. MLP is applied to encode the original feature vector of each process node to obtain the process node feature embedding. After calculating the embedding of the process node and the machine node, the graph pooling vector is calculated. The calculation formula of the graph pooling vector is as follows: Where h t is the graph pooling vector; The process for the newly arrived workpiece; is the process set of the newly arrived workpiece; It is the set of processes that are not assigned to the current FMU; Embedding process node features; is the total number of machines; For machines The final embedding vector of .

8. The method for dynamic distributed flexible workshop scheduling based on hierarchical multi-agent deep reinforcement learning according to claim 7 is characterized in that: The underlying agent outputs all feasible OM pair probabilities through the policy network, adopts sampling or greedy decoding strategy, and selects actions according to the obtained distribution. The calculation formula of OM pair probability is as follows: in, is the probability of action selection of the lower-level agent, and MLP(·) is a multi-layer perceptron.

9. The method for dynamic distributed flexible workshop scheduling based on hierarchical multi-agent deep reinforcement learning according to claim 8, characterized in that: The reward function in the underlying agent is as follows: Where, Indicates the maximum completion time of flexible manufacturing unit l at a certain moment; LB * (O ij ) represents the process O at a certain moment ij The scheduled completion time or the minimum estimated completion time of the unscheduled operations.

10. The method for dynamic distributed flexible workshop scheduling based on hierarchical multi-agent deep reinforcement learning according to claim 1, characterized in that: Both layers of agents are trained using a proximal policy optimization algorithm; a staged training procedure is adopted: the bottom-level agent is first trained on fixed-size instances, and the upper-level agent is subsequently trained using the policy derived from the bottom-level agent, focusing on global artifact distribution.

Images