Dynamic integrated process planning and scheduling method based on graph reinforcement learning
By constructing dynamic heterogeneous graphs and heterogeneous graph Actor-Critic networks, the challenges of process selection, manufacturing cell allocation, and worker team selection in reconfigurable manufacturing workshops are solved, achieving efficient production scheduling and improving production efficiency and resource utilization.
Patent Information
- Application Number
- CN202611322032.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-28
- Publication Date
- 2026-11-06
AI Technical Summary
In reconfigurable matrix manufacturing workshops, existing technologies struggle to effectively coordinate process selection, manufacturing cell allocation, and worker team selection, leading to low production efficiency and low resource utilization. In particular, scheduling decisions are difficult when manufacturing cells malfunction or workers have varying skill levels.
A dynamic integrated process planning and scheduling method based on graph reinforcement learning is adopted. By constructing a dynamic heterogeneous graph, a real-time scheduling strategy is generated using a heterogeneous graph Actor-Critic network. Combined with an event-driven scheduling mechanism and imitation-enhanced proximal strategy optimization training, efficient collaboration among processes, manufacturing units, and worker teams is achieved.
It improves the responsiveness and resource utilization of the production process, reduces the total weighted delay, and enhances the production efficiency and resource utilization efficiency of the reconfigurable manufacturing workshop.
Smart Images
Figure CN123051003A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent manufacturing technology, specifically a dynamic integrated process planning and scheduling method based on graph reinforcement learning. Background Technology
[0002] A reconfigurable matrix manufacturing workshop consists of multiple reconfigurable manufacturing cells, a number of workers with different roles and skill levels, and production operations that are dynamically released or arrived over time. The reconfigurable manufacturing cells are not static resources with fixed processing capabilities, but can switch between different capability states according to production task requirements. Workers are not merely auxiliary resources of the manufacturing cells, but rather teams of workers fulfilling specific roles and skill requirements, working together with the manufacturing cells to complete process execution. Based on these characteristics, this type of problem can be abstracted as a dynamic integrated process planning and scheduling problem considering the collaboration between reconfigurable manufacturing cells and workers.
[0003] In actual production, when scheduling begins, some jobs are usually already waiting to be processed, and subsequent jobs will arrive one after another. Each job consists of several processes, and there are sequential constraints between the processes; a certain process can only enter the candidate set after all its preceding processes are completed; at the same time, the manufacturing unit with the corresponding capabilities and the team of workers who meet the job and skill requirements must be available before the process can be scheduled for execution.
[0004] Reconfigurable manufacturing cells can be configured with one or more manufacturing capabilities. When the capability required for the next task is inconsistent with the current capability of the manufacturing cell, a capability switch is required, resulting in corresponding reconfiguration or preparation time. Manufacturing cell failures may also occur during production. When a failure occurs, the task being processed is interrupted; after repair, the manufacturing cell becomes available again, and unfinished tasks are re-entered into the scheduling process based on the current shop floor status.
[0005] Workforce resources vary in job roles and skill levels. For processes requiring specific roles, the work can be completed by a single worker or by multiple workers with the same role working collaboratively. Different skill levels affect actual processing efficiency and subsequent resource utilization. Therefore, scheduling decisions cannot simply determine "which process to do first and which manufacturing unit to use," but also need to determine the selection method for worker teams, striking a balance between processing efficiency, retention of highly skilled workers, and workload balancing. Summary of the Invention
[0006] To address the shortcomings of existing technologies, the purpose of this invention is to provide a dynamic integrated process planning and scheduling method based on graph reinforcement learning.
[0007] The technical solution adopted by this invention to solve its technical problem is:
[0008] A dynamic ensemble process planning and scheduling method based on graph reinforcement learning includes the following steps:
[0009] S1. Initialize the workshop environment, read initial operation information, reconfigurable manufacturing cell capacity configuration and reconfiguration time information, worker job roles and skill level information;
[0010] S2. When a scheduling event is detected, a scheduling decision is triggered; scheduling events include the arrival of a new job, completion of a process, failure of a reconfigurable manufacturing cell, and completion of maintenance of a reconfigurable manufacturing cell.
[0011] S3. Based on the incomplete tasks, available reconfigurable manufacturing cells, and available workers at the current moment, construct a dynamic heterogeneous graph. And based on dynamic heterogeneous graph Generate a set of candidate actions that satisfy the constraints. ;
[0012] Among them, dynamic heterogeneous graph Node types include job nodes, process nodes, reconfigurable manufacturing cell nodes, and worker nodes, in a dynamic heterogeneous graph. The edge types include process priority relationship edges, process and reconfigurable manufacturing cell process and process and worker role matching relationship edges. The edge attributes include processing time, capability switching preparation time and skill efficiency coefficient.
[0013] S4. Dynamic heterogeneous graph and candidate action set Input the pre-trained heterogeneous graph Actor-Critic network to obtain the probability score of each candidate action, and select the candidate action with the highest probability score as the current scheduling action;
[0014] The current scheduling action is represented by a quadruple. , Indicates the selected assignment. Indicate homework The processing steps that meet the priority constraints in the middle. This refers to a reconfigurable manufacturing cell that has processing capabilities and is currently available. This indicates the strategy chosen by the worker team;
[0015] S5. Execute the current scheduling action, and move the process... Assigned to reconfigurable manufacturing units And select strategies based on worker teams. Determine the participating processes The processing team;
[0016] S6. Update the workshop production status, including updating the job progress, the capacity configuration and available time of the reconfigurable manufacturing cells, the worker occupancy status, and the fault and maintenance status of the reconfigurable manufacturing cells; and return to S2 until all jobs are completed.
[0017] As a preferred embodiment, a further technical solution of the present invention is:
[0018] Preferably, a dynamic heterogeneous graph is constructed in S3. The specific process includes:
[0019] S3-1. Obtain the set of all released but not yet completed jobs at the current time t. and the set of unfinished processes in each operation. ;
[0020] S3-2. Create a process node for each incomplete process, a job node for each incomplete task, a reconfigurable manufacturing cell node for each reconfigurable manufacturing cell, a worker node for each available worker, and also create virtual nodes representing the start and end of the process route.
[0021] S3-3. Establish process priority relationship edges: For any two processes within the same operation... and ,like yes The preceding process is established from point to The directed edges;
[0022] S3-4. Establish the processability relationship edge between the process and the reconfigurable manufacturing cell: If the process Required manufacturing capabilities and reconfigurable manufacturing cells It matches the current capability configuration, and If the current state is available, then create a processable relation edge and record the processing time in the edge attributes. and capability switching preparation time If the manufacturing unit is reconfigurable Current capabilities and processes If the required capabilities are the same, then ,otherwise The value represents the preparation time for switching from the current capability to the desired capability;
[0023] S3-5. Establish the role matching relationship between processes and workers. Edge: If the worker The job role meets the process requirements Required roles and workers If the current state is available, establish a role matching relationship edge and record the worker in the edge attribute. skill level and cumulative workload ;
[0024] S3-6. Employ a dynamic pruning mechanism to remove completed process nodes, completed operation nodes, reconfigurable manufacturing cell nodes that are in a faulty state and have not been repaired, and currently unreachable relation edges from the current dynamic heterogeneous graph.
[0025] Preferably, a candidate action set is generated in S3. The process specifically includes:
[0026] S3-7. Traverse all incomplete processes and select incomplete processes that have been completed by all preceding processes as schedulable processes.
[0027] S3-8. For each schedulable process, find all reconfigurable manufacturing unit nodes in the dynamic heterogeneous graph that have a processable relationship edge with the schedulable process node. Each found reconfigurable manufacturing unit node and its corresponding schedulable process constitute a schedulable process-reconfigurable manufacturing unit pair.
[0028] S3-9. For each schedulable process-reconfigurable manufacturing cell pair, find all available worker nodes in the dynamic heterogeneous graph that have role matching relationships with the schedulable process node; and for each type of worker team, select a strategy and generate a candidate action quadruple, and add it to the candidate action set.
[0029] The worker team's strategy selection strategy should include at least the following types:
[0030] Fastest strategy: Prioritize workers with the highest skill level currently available;
[0031] Balanced strategy: Prioritize workers whose skill level is close to the median level of available workers;
[0032] Conservative strategy: Prioritize workers whose skill level is below the median but who meet the requirements;
[0033] Minimum skill waste strategy: Select workers with the lowest skill level while meeting process requirements, and avoid having the cumulative workload of the lowest skill workers fall below the work limit.
[0034] Load balancing strategy: Prioritize workers with low cumulative workload.
[0035] Preferably, the heterogeneous graph Actor-Critic network in S4 includes a heterogeneous graph encoder, a policy network, and a value network;
[0036] The heterogeneous graph encoder maps the original features of job nodes, process nodes, reconfigurable manufacturing unit nodes, and worker nodes to a unified embedding space through a linear mapping layer, and then aggregates neighborhood information of various node embeddings through multi-layer heterogeneous graph convolution. Among them, for edges with edge attribute information, the heterogeneous graph encoder aggregates the edge attributes together with the source node embedding and the target node embedding in the edge-aware fusion function in each graph convolution layer. After each graph convolution layer, layer normalization and residual connection are performed.
[0037] The policy network, based on the final embedding of the job node, process node and reconfigurable manufacturing unit node corresponding to the candidate action, splices the edge attributes between the process and the reconfigurable manufacturing unit, the matching statistical features between the process and the worker, and the global state embedding, and outputs the score of the candidate action through a multi-layer fully connected network. The scores of all candidate actions are normalized by the Softmax function to obtain the action probability distribution.
[0038] The value network pools the final embeddings of all nodes in the dynamic heterogeneous graph to obtain graph-level embeddings. After concatenating the graph-level embeddings with the global state embeddings, it outputs the value estimate of the current state through a multi-layer fully connected network.
[0039] The preferred training process for a heterogeneous graph Actor-Critic network specifically includes:
[0040] T1. Constructing an expert dataset Run candidate heuristic rules on multiple training environment instances, select the rule with the smallest final total weighted delay in each instance as the expert rule, and use the state-action pairs in the scheduling trajectory generated by the expert rule. Store in the expert dataset;
[0041] T2. Pre-train the policy network of the heterogeneous graph Actor-Critic network using behavior cloning with an expert dataset, and minimize the behavior cloning loss function. Update network parameters:
[0042] ;
[0043] in, Indicates the policy network in state Select action The probability of;
[0044] T3. Perform imitation-enhanced proximal policy optimization training in a simulation environment; each training iteration includes the following sub-steps:
[0045] T3-1, Use the current strategy In the training environment, scheduling trajectories are collected, and the state at each decision moment in the trajectory is recorded. ,action Instant rewards State value estimation and termination mark And store it in the trajectory cache area. ;
[0046] T3-2, Record-Based Instant Rewards and state value estimation Calculate the cumulative discounted return for each time step in each trajectory. and dominance function State value estimation during the calculation process Used as a baseline to reduce the variance of the policy gradient:
[0047] ;
[0048] ;
[0049] in, This represents the immediate reward at the k-th step after the current decision time t. As a discount factor, For GAE parameters, = The time-series differential error is T, where T is the trajectory termination time step.
[0050] T3-3. Perform multiple rounds of small-batch processing on the data in the trajectory buffer. Update, using the calculated advantage function The parameter update direction of the policy network is guided; for each update round, the pruning objective function of the policy network is calculated. Value function loss Entropy regularization term and imitation learning loss :
[0051] ;
[0052] in, The importance sampling ratio, This is the cutting factor;
[0053] ;
[0054] ;
[0055] ;
[0056] T3-4. Minimize the joint loss function to update network parameters:
[0057] ;
[0058] in, The entropy regularity coefficient; This is the value loss coefficient; To mimic the learning coefficient, the coefficient is dynamically decayed according to a preset strategy during training, so that the policy network mainly imitates the behavior of experts in the early stage of training, and gradually transitions to autonomous exploration and optimization dominated by reinforcement learning reward signals as training progresses.
[0059] T4. Repeat step T3 until the preset number of training rounds is reached or the policy performance converges, and output the parameters of the trained heterogeneous graph Actor-Critic network.
[0060] Preferred, instant rewards Constructed based on the estimated change in total weighted tardiness:
[0061] ;
[0062] in, Represents the state at decision time t. Below is the total weighted delinquency estimate for all unfinished tasks; This represents the state at the next decision time t+1 after the action is executed. Below, the total weighted delay is re-estimated;
[0063] Estimated total weighted delay for:
[0064] ;
[0065] in, For homework Priority weights; For homework Delivery period; This is the set of jobs that have been released but not yet completed in the current state; For homework In state The estimated completion time is as follows.
[0066] Preferably, for any unfinished task In state The estimated completion time Calculated in the following way:
[0067] ;
[0068] in, Indicate homework A collection of processes that have not yet been completed;
[0069] Determine as follows:
[0070] If the process Currently being processed, then The value represents the available time allocated to the reconfigurable manufacturing cell for that process. This refers to the estimated completion time of the process.
[0071] If the process It is ready but has not yet started processing. Takes the earliest completion time among all reconfigurable manufacturing cells capable of performing this operation. This refers to the expected start time of the process.
[0072] If the process If it is not ready yet, then The value is taken as the current time t;
[0073] Tail processing time Indicate process The sum of the remaining processing times on the longest path in the process tree is calculated recursively:
[0074] ;
[0075] in, Indicate process The set of all direct successor processes, For process The estimated processing time; Indicates subsequent processes The longest tail processing time; for the last process without subsequent processes, .
[0076] Preferably, updating the workshop production status in S6 includes:
[0077] S6-1. If the process is completed, update the completion progress of the job, mark the process as completed, and release the occupied reconfigurable manufacturing cell and the occupied worker team.
[0078] S6-2. If a reconfigurable manufacturing cell fails during the processing of a process, the process is interrupted, the process is remarked as incomplete, the reconfigurable manufacturing cell is marked as faulty, and a maintenance timer is started; after maintenance is completed, the reconfigurable manufacturing cell is marked as available.
[0079] S6-3. If a new task arrives, add the new task's process to the set of incomplete processes and establish the corresponding process priority edge according to the process priority relationship.
[0080] S6-4. Update the available time of all reconfigurable manufacturing cells, the available time and cumulative load of all workers, and use the updated shop floor state as the initial state for the next decision moment.
[0081] Preferably, a candidate action set is generated in S3. The process satisfies the following constraints:
[0082] The team of workers participating in the processing procedure must meet the job roles and number of people required by the procedure; otherwise, the candidate action corresponding to the procedure will be invalid.
[0083] The same worker cannot be assigned to multiple work processes at the same time;
[0084] When performing scheduling actions in S5, the following constraints must be met:
[0085] Each process can be assigned to at most one reconfigurable manufacturing cell at any given time;
[0086] Each reconfigurable manufacturing cell can be assigned at most one process at any given time;
[0087] The processing of a process cannot be manually split or paused, except for interruptions caused by a fault in the reconfigurable manufacturing unit;
[0088] When a reconfigurable manufacturing unit switches between different capability states, it needs to consume reconfiguration preparation time related to the capability switching sequence in the edge attributes.
[0089] This invention also discloses a dynamic integrated process planning and scheduling system based on graph reinforcement learning, specifically including:
[0090] The data acquisition module is used to initialize the workshop environment and read initial operation information, reconfigurable manufacturing cell capacity configuration and reconfiguration time information, and worker job roles and skill level information.
[0091] The event monitoring module is used to trigger scheduling decisions when a scheduling event is detected. Scheduling events include the arrival of a new job, completion of a process, failure of a reconfigurable manufacturing cell, and completion of maintenance of a reconfigurable manufacturing cell.
[0092] The candidate action generation module is used to construct a dynamic heterogeneous graph based on the currently incomplete tasks, available reconfigurable manufacturing cells, and available workers. And based on dynamic heterogeneous graph Generate a set of candidate actions that satisfy the constraints. ;
[0093] The scheduling decision module is used to process dynamic heterogeneous graphs. and candidate action set Input the pre-trained heterogeneous graph Actor-Critic network to obtain the probability score of each candidate action, and select the candidate action with the highest probability score as the current scheduling action;
[0094] The action execution module is used to execute the currently scheduled action, and to process the work process. Assigned to reconfigurable manufacturing units And select strategies based on worker teams. Determine the participating processes Processing worker team
[0095] The status update module is used to update the workshop production status, including updating the job progress, the capacity configuration and available time of the reconfigurable manufacturing unit, the worker occupancy status, and the fault and maintenance status of the reconfigurable manufacturing unit; the trigger event monitoring module continues to monitor the next scheduling event until all jobs are completed.
[0096] The present invention, which adopts the above technical solution, has the following prominent features compared with the prior art:
[0097] Jobs, processes, reconfigurable manufacturing cells, and workers are represented as different types of graph nodes. Process priority relationships, process-to-manufacturability relationships, process-to-worker role matching relationships, and resource availability status are represented as heterogeneous edges and their attributes. In the event-driven scheduling process, a dynamic heterogeneous graph is generated based on the current shop floor state, forming a state input that includes node features, edge features, global features, and actionable information. The heterogeneous graph Actor-Critic network then provides a real-time scheduling strategy.
[0098] A reward signal is constructed by estimating changes in total weighted delay, and combined with heuristic expert data, behavioral cloning pre-training, and imitation-enhanced proximal policy optimization training methods to improve policy learning efficiency. In a frequently disturbed manufacturing environment, collaborative process selection, manufacturing cell allocation, and worker team selection are achieved, reducing total weighted delay and improving the response speed and resource utilization of the reconfigurable manufacturing shop. Attached Figure Description
[0099] Figure 1 This is a schematic diagram of the overall framework of the method in the embodiments of the present invention;
[0100] Figure 2 This is a schematic diagram of the method flow in an embodiment of the present invention;
[0101] Figure 3 This is a schematic diagram of the dynamic heterogeneity diagram in an embodiment of the present invention;
[0102] Figure 4 This is a schematic diagram of the system structure in an embodiment of the present invention. Detailed Implementation
[0103] The present invention will be further illustrated below with reference to specific embodiments. The purpose of this illustration is solely to provide a better understanding of the invention. Therefore, the examples given do not limit the scope of protection of the present invention.
[0104] This invention treats the dynamic integration process planning and scheduling problem as an event-driven sequential decision problem. The overall framework of this invention is as follows: Figure 1 As shown, when an event requiring scheduling occurs, the current status of the job, process, manufacturing cell, and worker is first collected; then a dynamic heterogeneous graph is constructed, and candidate actions that meet the constraints are generated; subsequently, the graph reinforcement learning policy network scores the candidate actions and selects the scheduling action composed of the selection rules of the job, process, manufacturing cell, and worker team; after the action is executed, the shop floor status is updated accordingly.
[0105] Traditional scheduling methods often treat machines or manufacturing units as single resources, making it difficult to simultaneously represent process relationships, capacity switching, and worker collaboration. This invention employs dynamic heterogeneous graphs to describe multiple types of production entities and their interrelationships. This not only characterizes process priority relationships, manufacturing unit processing capabilities, and worker job matching relationships, but also incorporates worker skill levels, the current capacity status of manufacturing units, and dynamic perturbation information into the input of reinforcement learning strategies.
[0106] Specifically, such as Figure 2 As shown in the figure, this embodiment presents a dynamic integrated process planning and scheduling method based on graph reinforcement learning, including the following steps:
[0107] S1. Initialize the workshop environment, read initial operation information, reconfigurable manufacturing cell capacity configuration and reconfiguration time information, worker job roles and skill level information;
[0108] S2. When a scheduling event is detected, a scheduling decision is triggered; scheduling events include the arrival of a new job, completion of a process, failure of a reconfigurable manufacturing cell, and completion of maintenance of a reconfigurable manufacturing cell.
[0109] S3. Based on the incomplete tasks, available reconfigurable manufacturing cells, and available workers at the current moment, construct a dynamic heterogeneous graph. And based on dynamic heterogeneous graph Generate a set of candidate actions that satisfy the constraints. ;
[0110] Among them, dynamic heterogeneous graph Node types include job nodes, process nodes, reconfigurable manufacturing cell nodes, and worker nodes, in a dynamic heterogeneous graph. The edge types include process priority relationship edges, process and reconfigurable manufacturing cell process and process and worker role matching relationship edges. The edge attributes include processing time, capability switching preparation time and skill efficiency coefficient.
[0111] S4. Dynamic heterogeneous graph and candidate action set Input the pre-trained heterogeneous graph Actor-Critic network to obtain the probability score of each candidate action, and select the candidate action with the highest probability score as the current scheduling action;
[0112] The current scheduling action is represented by a quadruple. , Indicates the selected assignment. Indicate homework The processing steps that meet the priority constraints in the middle. This refers to a reconfigurable manufacturing cell that has processing capabilities and is currently available. The action space consists of a selection strategy for the worker team; each action includes the selected job, the processing steps in the job that satisfy the priority constraints, the manufacturing cell that has processing capabilities and is currently available, and the selection strategy for determining the worker team. This invention does not directly enumerate all feasible worker combinations, but determines them through the selection strategy for the worker team. This reduces the size of the action space while still retaining the ability to describe the collaborative execution of workers.
[0113] S5. Execute the current scheduling action, and move the process... Assigned to reconfigurable manufacturing units And select strategies based on worker teams. Determine the participating processes The processing team;
[0114] S6. Update the workshop production status, including updating the job progress, the capacity configuration and available time of the reconfigurable manufacturing cells, the worker occupancy status, and the fault and maintenance status of the reconfigurable manufacturing cells; and return to S2 until all jobs are completed.
[0115] In implementation, at the current moment of a scheduling event, a dynamic heterogeneous graph is constructed. The nodes in the graph include job nodes, process nodes, reconfigurable manufacturing cell nodes, worker nodes, and virtual nodes representing the start and end points of the process route. Job nodes describe release time, delivery date, priority weight, number of remaining processes, and completion progress. Process nodes describe required manufacturing capacity, required job roles, estimated processing time, completion status of predecessors, and current schedulable status. Manufacturing cell nodes describe current capacity configuration, available time, fault repair status, queue load, and reconfiguration cost. Worker nodes describe job roles, skill levels, available time, cumulative load, and current occupancy status. Edges in the dynamic heterogeneous graph include job-process affiliation edges, process priority edges, process-manufacturing cell processability edges, process-worker role matching edges, and corresponding reverse edges. For relationships with resource attributes, edge features are also set, such as the processing time of a manufacturing cell executing a certain process, capacity switching preparation time, worker efficiency coefficient when participating in the process, and estimated start time.
[0116] Specifically, constructing dynamic heterogeneous graphs The specific process includes:
[0117] S3-1. Obtain the set of all released but not yet completed jobs at the current time t. and the set of unfinished processes in each operation. ;
[0118] S3-2, Create a process node for each incomplete process. Create a job node for each incomplete job and a reconfigurable manufacturing cell node for each reconfigurable manufacturing cell. Create a worker node for each available worker. Simultaneously, virtual nodes representing the start and end of the process route are created;
[0119] S3-3. Establish process priority relationship edges: For any two processes within the same operation... and ,like yes The preceding process is established from point to The directed edges;
[0120] S3-4. Establish the processability relationship edge between the process and the reconfigurable manufacturing cell: If the process Required manufacturing capabilities and reconfigurable manufacturing cells It matches the current capability configuration, and If the current state is available, then create a processable relation edge and record the processing time in the edge attributes. and capability switching preparation time If the manufacturing unit is reconfigurable Current capabilities and processes If the required capabilities are the same, then ,otherwise The value represents the preparation time for switching from the current capability to the desired capability;
[0121] S3-5. Establish the role matching relationship between processes and workers. Edge: If the worker The job role meets the process requirements Required roles and workers If the current state is available, establish a role matching relationship edge and record the worker in the edge attribute. skill level and cumulative workload ;
[0122] S3-6. Employ a dynamic pruning mechanism to remove completed process nodes, completed operation nodes, reconfigurable manufacturing cell nodes that are in a faulty state and have not been repaired, and currently unreachable relation edges from the current dynamic heterogeneous graph.
[0123] like Figure 3 As shown, where Figure 3 a is a schematic diagram of a dynamic heterogeneous diagram provided in an embodiment of the present invention. In the diagram, the black line represents the process priority relationship, the yellow dashed line represents the feasible process-manufacturing unit relationship, and the green dashed line represents the feasible process-worker relationship. Figure 3 b is an example of a selected process-resource allocation relationship (current scheduling action).
[0124] To improve real-time decision-making efficiency, this invention employs a dynamic pruning mechanism. At each decision moment, only released but not yet completed jobs and incomplete procedures are retained; completed nodes and temporarily unreachable relationships are removed from the current graph, thereby reducing invalid searches and improving the speed of strategy reasoning.
[0125] During implementation, a set of candidate actions is generated. The process specifically includes:
[0126] S3-7. Traverse all incomplete processes and select incomplete processes that have been completed by all preceding processes as schedulable processes.
[0127] S3-8. For each schedulable process, find all reconfigurable manufacturing unit nodes in the dynamic heterogeneous graph that have a processable relationship edge with the schedulable process node. Each found reconfigurable manufacturing unit node and its corresponding schedulable process constitute a schedulable process-reconfigurable manufacturing unit pair.
[0128] S3-9. For each schedulable process-reconfigurable manufacturing cell pair, find all available worker nodes in the dynamic heterogeneous graph that have role matching relationships with the schedulable process node; and for each type of worker team, select a strategy and generate a candidate action quadruple, and add it to the candidate action set.
[0129] The worker team's strategy selection strategy should include at least the following types:
[0130] Fastest strategy: Prioritize workers with the highest skill level currently available;
[0131] Balanced strategy: Prioritize workers whose skill level is close to the median level of available workers;
[0132] Conservative strategy: Prioritize workers whose skill level is below the median but who meet the requirements;
[0133] Minimum skill waste strategy: Select workers with the lowest skill level while meeting process requirements, and avoid having the cumulative workload of the lowest skill workers fall below the work limit.
[0134] Load balancing strategy: Prioritize workers with low cumulative workload.
[0135] This invention constructs a heterogeneous graph Actor-Critic network, employing an approximate model of policy and value functions. The network first encodes job nodes, process nodes, manufacturing unit nodes, worker nodes, edge attributes, and global features, mapping different types of original features to a unified space. Then, it aggregates neighborhood information across different relationships through multi-layer heterogeneous graph convolution, enabling node representations to simultaneously integrate process priority relationships, resource processing relationships, and worker matching relationships.
[0136] For topological relationships without continuous edge attributes, the network updates node representations using a graph convolution method based on neighborhood aggregation. For resource relationships with attributes such as processing time, preparation time, and skill efficiency, the network inputs edge attributes along with the source and target node representations into the edge-aware update module to enhance the representation of resource adaptability. Normalization, nonlinear activation, and residual connections are applied after each graph convolution layer to improve the training stability of deep networks.
[0137] The policy network extracts the node embeddings of the corresponding job, process, and manufacturing unit for each candidate action, and concatenates the process-manufacturing unit edge features, process-worker matching statistical features, and global state embeddings to output the score of the candidate action; after normalization, the action probability distribution is obtained. The value network pools the various node embeddings and then concatenates them with the global features to output the value estimate of the current state.
[0138] Specifically, the heterogeneous graph Actor-Critic network includes a heterogeneous graph encoder, a policy network, and a value network;
[0139] The heterogeneous graph encoder maps the original features of job nodes, process nodes, reconfigurable manufacturing unit nodes, and worker nodes to a unified embedding space through a linear mapping layer, and then aggregates neighborhood information of various node embeddings through multi-layer heterogeneous graph convolution. Among them, for edges with edge attribute information, the heterogeneous graph encoder aggregates the edge attributes together with the source node embedding and the target node embedding in the edge-aware fusion function in each graph convolution layer. After each graph convolution layer, layer normalization and residual connection are performed.
[0140] The policy network, based on the final embeddings of the job node, process node, and reconfigurable manufacturing unit node corresponding to the candidate action, concatenates the edge attributes between the process and the reconfigurable manufacturing unit, the matching statistical features between the process and the worker, and the global state embedding. It outputs the score of the candidate action through a multi-layer fully connected network. The scores of all candidate actions are normalized by the Softmax function to obtain the action probability distribution. The output of the shared heterogeneous graph encoder is used to extract the embedding of the corresponding node for each candidate action. After concatenating the edge attributes and global features, the score of the action is output through MLP.
[0141] The value network pools the final embeddings of all nodes in the dynamic heterogeneous graph to obtain graph-level embeddings. These graph-level embeddings are then concatenated with the global state embeddings and output as a value estimate of the current state through a multi-layer fully connected network. The output of the shared heterogeneous graph encoder is used to pool the embeddings of all nodes to obtain graph-level embeddings. These are then concatenated with global features and output as a value estimate of the current state through an MLP. The global state embedding is encoded from global state features, which include the current simulation time, shop floor system load, number of unfinished tasks, manufacturing cell failure rate, average resource utilization, and average task urgency.
[0142] In practice, the training process of the heterogeneous graph Actor-Critic network specifically includes:
[0143] T1. Constructing an expert dataset Run candidate heuristic rules on multiple training environment instances, select the rule with the smallest final total weighted delay in each instance as the expert rule, and use the state-action pairs in the scheduling trajectory generated by the expert rule. Store in the expert dataset;
[0144] T2. Pre-train the policy network of the heterogeneous graph Actor-Critic network using behavior cloning with an expert dataset, and minimize the behavior cloning loss function. Update network parameters:
[0145] ;
[0146] in, Indicates the policy network in state Select action The probability of;
[0147] T3. Perform imitation-enhanced proximal policy optimization training in a simulation environment; each training iteration includes the following sub-steps:
[0148] T3-1, Use the current strategy In the training environment, scheduling trajectories are collected, and the state at each decision moment in the trajectory is recorded. ,action Instant rewards State value estimation and termination mark And store it in the trajectory cache area. ;
[0149] T3-2, Record-Based Instant Rewards and state value estimation Calculate the cumulative discounted return for each time step in each trajectory. and dominance function State value estimation during the calculation process Used as a baseline to reduce the variance of the policy gradient:
[0150] ;
[0151] ;
[0152] in, This represents the immediate reward at the k-th step after the current decision time t. As a discount factor, For GAE parameters, = The time-series differential error is T, where T is the trajectory termination time step.
[0153] T3-3. Perform multiple rounds of small-batch processing on the data in the trajectory buffer. Update, using the calculated advantage function The parameter update direction of the policy network is guided; for each update round, the pruning objective function of the policy network is calculated. Value function loss Entropy regularization term and imitation learning loss :
[0154] ;
[0155] in, The importance sampling ratio, This is the cutting factor;
[0156] ;
[0157] ;
[0158] ;
[0159] T3-4. Minimize the joint loss function to update network parameters:
[0160] ;
[0161] in, The entropy regularity coefficient; This is the value loss coefficient; To mimic the learning coefficient, the coefficient is dynamically decayed according to a preset strategy during training, so that the policy network mainly imitates the behavior of experts in the early stage of training, and gradually transitions to autonomous exploration and optimization dominated by reinforcement learning reward signals as training progresses.
[0162] T4. Repeat step T3 until the preset number of training rounds is reached or the policy performance converges, and output the parameters of the trained heterogeneous graph Actor-Critic network.
[0163] In-time rewards during implementation Constructed based on the estimated change in total weighted tardiness:
[0164] ;
[0165] in, Represents the state at decision time t. Below is the total weighted delinquency estimate for all unfinished tasks; This represents the state at the next decision time t+1 after the action is executed. Below, the total weighted delay is re-estimated;
[0166] Estimated total weighted delay for:
[0167] ;
[0168] in, For homework Priority weights; For homework Delivery period; This is the set of jobs that have been released but not yet completed in the current state; For homework In state The estimated completion time is as follows.
[0169] During implementation, for any unfinished task In state The estimated completion time Calculated in the following way:
[0170] ;
[0171] in, Indicate homework A collection of processes that have not yet been completed;
[0172] Determine as follows:
[0173] If the process Currently being processed, then The value represents the available time allocated to the reconfigurable manufacturing cell for that process. This refers to the estimated completion time of the process.
[0174] If the process It is ready but has not yet started processing. Takes the earliest completion time among all reconfigurable manufacturing cells capable of performing this operation. This refers to the expected start time of the process.
[0175] If the process If it is not ready yet, then The value is taken as the current time t;
[0176] Tail processing time Indicate process The sum of the remaining processing times on the longest path in the process tree is calculated recursively:
[0177] ;
[0178] in, Indicate process The set of all direct successor processes, For process The estimated processing time; Indicates subsequent processes The longest tail processing time; for the last process without subsequent processes, .
[0179] During implementation, S6 updates the workshop production status, specifically including:
[0180] S6-1. If the process is completed, update the completion progress of the job, mark the process as completed, and release the occupied reconfigurable manufacturing cell and the occupied worker team.
[0181] S6-2. If a reconfigurable manufacturing cell fails during the processing of a process, the process is interrupted, the process is remarked as incomplete, the reconfigurable manufacturing cell is marked as faulty, and a maintenance timer is started; after maintenance is completed, the reconfigurable manufacturing cell is marked as available.
[0182] S6-3. If a new task arrives, add the new task's process to the set of incomplete processes and establish the corresponding process priority edge according to the process priority relationship.
[0183] S6-4. Update the available time of all reconfigurable manufacturing cells, the available time and cumulative load of all workers, and use the updated shop floor state as the initial state for the next decision moment.
[0184] During implementation, a set of candidate actions is generated. The process satisfies the following constraints:
[0185] The team of workers participating in the processing procedure must meet the job roles and number of people required by the procedure; otherwise, the candidate action corresponding to the procedure will be invalid.
[0186] The same worker cannot be assigned to multiple work processes at the same time;
[0187] When performing scheduling actions in S5, the following constraints must be met:
[0188] Each process can be assigned to at most one reconfigurable manufacturing cell at any given time;
[0189] Each reconfigurable manufacturing cell can be assigned at most one process at any given time;
[0190] The processing of a process cannot be manually split or paused, except for interruptions caused by a fault in the reconfigurable manufacturing unit;
[0191] When a reconfigurable manufacturing unit switches between different capability states, it needs to consume reconfiguration preparation time related to the capability switching sequence in the edge attributes.
[0192] See Figure 4 The present invention also discloses a dynamic integrated process planning and scheduling system based on graph reinforcement learning, specifically including:
[0193] The data acquisition module is used to initialize the workshop environment and read initial operation information, reconfigurable manufacturing cell capacity configuration and reconfiguration time information, and worker job roles and skill level information.
[0194] The event monitoring module is used to trigger scheduling decisions when a scheduling event is detected. Scheduling events include the arrival of a new job, completion of a process, failure of a reconfigurable manufacturing cell, and completion of maintenance of a reconfigurable manufacturing cell.
[0195] The candidate action generation module is used to construct a dynamic heterogeneous graph based on the currently incomplete tasks, available reconfigurable manufacturing cells, and available workers. And based on dynamic heterogeneous graph Generate a set of candidate actions that satisfy the constraints. ;
[0196] The scheduling decision module is used to process dynamic heterogeneous graphs. and candidate action set Input the pre-trained heterogeneous graph Actor-Critic network to obtain the probability score of each candidate action, and select the candidate action with the highest probability score as the current scheduling action;
[0197] The action execution module is used to execute the currently scheduled action, and to process the work process. Assigned to reconfigurable manufacturing units And select strategies based on worker teams. Determine the participating processes Processing worker team
[0198] The status update module is used to update the workshop production status, including updating the job progress, the capacity configuration and available time of the reconfigurable manufacturing unit, the worker occupancy status, and the fault and maintenance status of the reconfigurable manufacturing unit; the trigger event monitoring module continues to monitor the next scheduling event until all jobs are completed.
[0199] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of the invention. All equivalent changes made based on the description and drawings of the present invention are included within the scope of the present invention.
Claims
1. A dynamic integrated process planning and scheduling method based on graph reinforcement learning, characterized in that, Includes the following steps: S1. Initialize the workshop environment, read initial operation information, reconfigurable manufacturing cell capacity configuration and reconfiguration time information, worker job roles and skill level information; S2. When a scheduling event is detected, a scheduling decision is triggered; scheduling events include the arrival of a new job, completion of a process, failure of a reconfigurable manufacturing cell, and completion of maintenance of a reconfigurable manufacturing cell. S3. Based on the incomplete tasks, available reconfigurable manufacturing cells, and available workers at the current moment, construct a dynamic heterogeneous graph. And based on dynamic heterogeneous graph Generate a set of candidate actions that satisfy the constraints. ; Among them, dynamic heterogeneous graph Node types include job nodes, process nodes, reconfigurable manufacturing cell nodes, and worker nodes, in a dynamic heterogeneous graph. The edge types include process priority relationship edges, process and reconfigurable manufacturing cell process and process and worker role matching relationship edges. The edge attributes include processing time, capability switching preparation time and skill efficiency coefficient. S4. Dynamic heterogeneous graph and candidate action set Input the pre-trained heterogeneous graph Actor-Critic network to obtain the probability score of each candidate action, and select the candidate action with the highest probability score as the current scheduling action; The current scheduling action is represented by a quadruple. , Indicates the selected assignment. Indicate homework The processing steps that meet the priority constraints in the middle. This refers to a reconfigurable manufacturing cell that has processing capabilities and is currently available. This indicates the strategy chosen by the worker team; S5. Execute the current scheduling action, and move the process... Assigned to reconfigurable manufacturing units And select strategies based on worker teams. Determine the participating processes The processing team; S6. Update the workshop production status, including updating the job progress, the capacity configuration and available time of the reconfigurable manufacturing cells, the worker occupancy status, and the fault and maintenance status of the reconfigurable manufacturing cells; and return to S2 until all jobs are completed.
2. The dynamic ensemble process planning and scheduling method based on graph reinforcement learning according to claim 1, characterized in that, Constructing dynamic heterogeneous graphs in S3 The specific process includes: S3-1. Obtain the set of all released but not yet completed jobs at the current time t. and the set of unfinished processes in each operation. ; S3-2. Create a process node for each incomplete process, a job node for each incomplete task, a reconfigurable manufacturing cell node for each reconfigurable manufacturing cell, a worker node for each available worker, and also create virtual nodes representing the start and end of the process route. S3-3. Establish process priority relationship edges: For any two processes within the same operation... and ,like yes The preceding process is established from point to The directed edges; S3-4. Establish the processability relationship edge between the process and the reconfigurable manufacturing cell: If the process Required manufacturing capabilities and reconfigurable manufacturing cells It matches the current capability configuration, and If the current state is available, then create a processable relation edge and record the processing time in the edge attributes. and capability switching preparation time If the manufacturing unit is reconfigurable Current capabilities and processes If the required capabilities are the same, then ,otherwise The value represents the preparation time for switching from the current capability to the desired capability; S3-5. Establish the role matching relationship between processes and workers. Edge: If the worker The job role meets the process requirements Required roles and workers If the current state is available, establish a role matching relationship edge and record the worker in the edge attribute. skill level and cumulative workload ; S3-6. Employ a dynamic pruning mechanism to remove completed process nodes, completed operation nodes, reconfigurable manufacturing cell nodes that are in a faulty state and have not been repaired, and currently unreachable relation edges from the current dynamic heterogeneous graph.
3. The dynamic ensemble process planning and scheduling method based on graph reinforcement learning according to claim 2, characterized in that, Generate a set of candidate actions in S3 The process specifically includes: S3-7. Traverse all incomplete processes and select incomplete processes that have been completed by all preceding processes as schedulable processes. S3-8. For each schedulable process, find all reconfigurable manufacturing unit nodes in the dynamic heterogeneous graph that have a processable relationship edge with the schedulable process node. Each found reconfigurable manufacturing unit node and its corresponding schedulable process constitute a schedulable process-reconfigurable manufacturing unit pair. S3-9. For each schedulable process-reconfigurable manufacturing cell pair, find all available worker nodes in the dynamic heterogeneous graph that have role matching relationships with the schedulable process node; and for each type of worker team, select a strategy and generate a candidate action quadruple, and add it to the candidate action set. The worker team's strategy selection strategy should include at least the following types: Fastest strategy: Prioritize workers with the highest skill level currently available; Balanced strategy: Prioritize workers whose skill level is close to the median level of available workers; Conservative strategy: Prioritize workers whose skill level is below the median but who meet the requirements; Minimum skill waste strategy: Select workers with the lowest skill level while meeting process requirements, and avoid having the cumulative workload of the lowest skill workers fall below the work limit. Load balancing strategy: Prioritize workers with low cumulative workload.
4. The dynamic ensemble process planning and scheduling method based on graph reinforcement learning according to claim 3, characterized in that, The heterogeneous graph Actor-Critic network in S4 includes a heterogeneous graph encoder, a policy network, and a value network. The heterogeneous graph encoder maps the original features of job nodes, process nodes, reconfigurable manufacturing unit nodes, and worker nodes to a unified embedding space through a linear mapping layer, and then aggregates neighborhood information of various node embeddings through multi-layer heterogeneous graph convolution. Among them, for edges with edge attribute information, the heterogeneous graph encoder aggregates the edge attributes together with the source node embedding and the target node embedding in the edge-aware fusion function in each graph convolution layer. After each graph convolution layer, layer normalization and residual connection are performed. The policy network, based on the final embedding of the job node, process node and reconfigurable manufacturing unit node corresponding to the candidate action, splices the edge attributes between the process and the reconfigurable manufacturing unit, the matching statistical features between the process and the worker, and the global state embedding, and outputs the score of the candidate action through a multi-layer fully connected network. The scores of all candidate actions are normalized by the Softmax function to obtain the action probability distribution. The value network pools the final embeddings of all nodes in the dynamic heterogeneous graph to obtain graph-level embeddings. After concatenating the graph-level embeddings with the global state embeddings, it outputs the value estimate of the current state through a multi-layer fully connected network.
5. The dynamic ensemble process planning and scheduling method based on graph reinforcement learning according to claim 4, characterized in that, The training process of the heterogeneous graph Actor-Critic network specifically includes: T1. Constructing an expert dataset Run candidate heuristic rules on multiple training environment instances, select the rule with the smallest final total weighted delay in each instance as the expert rule, and use the state-action pairs in the scheduling trajectory generated by the expert rule. Store in the expert dataset; T2. Pre-train the policy network of the heterogeneous graph Actor-Critic network using behavior cloning with an expert dataset, and minimize the behavior cloning loss function. Update network parameters: ； in, Indicates the policy network in state Select action The probability of; T3. Perform imitation-enhanced proximal policy optimization training in a simulation environment; each training iteration includes the following sub-steps: T3-1, Use the current strategy In the training environment, scheduling trajectories are collected, and the state at each decision moment in the trajectory is recorded. ,action Instant rewards State value estimation and termination mark And store it in the trajectory cache area. ; T3-2, Record-Based Instant Rewards and state value estimation Calculate the cumulative discounted return for each time step in each trajectory. and dominance function State value estimation during the calculation process Used as a baseline to reduce the variance of the policy gradient: ； ； in, This represents the immediate reward at the k-th step after the current decision time t. As a discount factor, For GAE parameters, = The time-series differential error is T, where T is the trajectory termination time step. T3-3. Perform multiple rounds of small-batch processing on the data in the trajectory buffer. Update, using the calculated advantage function The parameter update direction of the policy network is guided; for each update round, the pruning objective function of the policy network is calculated. Value function loss Entropy regularization term and imitation learning loss : ； in, Importance sampling ratio, This is the cutting factor; ； ； ； T3-4. Minimize the joint loss function to update network parameters: ； in, The entropy regularity coefficient; This is the value loss coefficient; To mimic the learning coefficient, the coefficient is dynamically decayed according to a preset strategy during training, so that the policy network mainly imitates the behavior of experts in the early stage of training, and gradually transitions to autonomous exploration and optimization dominated by reinforcement learning reward signals as training progresses. T4. Repeat step T3 until the preset number of training rounds is reached or the policy performance converges, and output the parameters of the trained heterogeneous graph Actor-Critic network.
6. The dynamic ensemble process planning and scheduling method based on graph reinforcement learning according to claim 5, characterized in that, Instant rewards Constructed based on the estimated change in total weighted tardiness: ； in, Represents the state at decision time t. Below is the total weighted delinquency estimate for all unfinished tasks; This represents the state at the next decision time t+1 after the action is executed. Below, the total weighted delay is re-estimated; Estimated total weighted delay for: ； in, For homework Priority weights; For homework Delivery period; This is the set of jobs that have been released but not yet completed in the current state; For homework In state The estimated completion time is as follows.
7. The dynamic ensemble process planning and scheduling method based on graph reinforcement learning according to claim 6, characterized in that, For any incomplete task In state The estimated completion time Calculated in the following way: ； in, Indicate homework A collection of processes that have not yet been completed; Determine as follows: If the process Currently being processed, then The value represents the available time allocated to the reconfigurable manufacturing cell for that process. This refers to the estimated completion time of the process. If the process It is ready but has not yet started processing. Takes the earliest completion time among all reconfigurable manufacturing cells capable of performing this operation. This refers to the expected start time of the process. If the process If it is not ready yet, then The value is taken as the current time t; Tail processing time Indicate process The sum of the remaining processing times on the longest path in the process tree is calculated recursively: ； in, Indicate process The set of all direct successor processes, For process The estimated processing time; Indicates subsequent processes The longest tail processing time; for the last process without subsequent processes, .
8. The dynamic ensemble process planning and scheduling method based on graph reinforcement learning according to claim 2, characterized in that, In S6, updating the workshop production status specifically includes: S6-1. If the process is completed, update the completion progress of the job, mark the process as completed, and release the occupied reconfigurable manufacturing cell and the occupied worker team. S6-2. If a reconfigurable manufacturing cell fails during the processing of a process, the process is interrupted, the process is remarked as incomplete, the reconfigurable manufacturing cell is marked as faulty, and a maintenance timer is started; after maintenance is completed, the reconfigurable manufacturing cell is marked as available. S6-3. If a new task arrives, add the new task's process to the set of incomplete processes and establish the corresponding process priority edge according to the process priority relationship. S6-4. Update the available time of all reconfigurable manufacturing cells, the available time and cumulative load of all workers, and use the updated shop floor state as the initial state for the next decision moment.
9. The dynamic ensemble process planning and scheduling method based on graph reinforcement learning according to claim 1, characterized in that, Generate a set of candidate actions in S3 The process satisfies the following constraints: The team of workers participating in the processing procedure must meet the job roles and number of people required by the procedure; otherwise, the candidate action corresponding to the procedure will be invalid. The same worker cannot be assigned to multiple work processes at the same time; When performing scheduling actions in S5, the following constraints must be met: Each process can be assigned to at most one reconfigurable manufacturing cell at any given time; Each reconfigurable manufacturing cell can be assigned at most one process at any given time; The processing of a process cannot be manually split or paused, except for interruptions caused by a fault in the reconfigurable manufacturing unit; When a reconfigurable manufacturing unit switches between different capability states, it needs to consume reconfiguration preparation time related to the capability switching sequence in the edge attributes.
10. A dynamic integrated process planning and scheduling system based on graph reinforcement learning, characterized in that, The method described in any one of claims 1 to 9 is used to implement the method, specifically including: The data acquisition module is used to initialize the workshop environment and read initial operation information, reconfigurable manufacturing cell capacity configuration and reconfiguration time information, and worker job roles and skill level information. The event monitoring module is used to trigger scheduling decisions when a scheduling event is detected. Scheduling events include the arrival of a new job, completion of a process, failure of a reconfigurable manufacturing cell, and completion of maintenance of a reconfigurable manufacturing cell. The candidate action generation module is used to construct a dynamic heterogeneous graph based on the currently incomplete tasks, available reconfigurable manufacturing cells, and available workers. And based on dynamic heterogeneous graph Generate a set of candidate actions that satisfy the constraints. ; The scheduling decision module is used to process dynamic heterogeneous graphs. and candidate action set Input the pre-trained heterogeneous graph Actor-Critic network to obtain the probability score of each candidate action, and select the candidate action with the highest probability score as the current scheduling action; The action execution module is used to execute the currently scheduled action, which will then execute the work process. Assigned to reconfigurable manufacturing units And select strategies based on worker teams. Determine the participating processes Processing worker team The status update module is used to update the workshop production status, including updating the job progress, the capacity configuration and available time of the reconfigurable manufacturing unit, the worker occupancy status, and the fault and maintenance status of the reconfigurable manufacturing unit; the trigger event monitoring module continues to monitor the next scheduling event until all jobs are completed.