A Cooperative Scheduling Method for Multi-Task, Multi-Machine, and Multi-Crane in Flexible Workshops Based on Hierarchical Multi-Agent Deep Reinforcement Learning
By employing a hierarchical multi-agent deep reinforcement learning approach, the collaborative scheduling of multiple tasks, multiple machines, and multiple overhead cranes in flexible workshops is optimized. This addresses the problem of insufficient coordination between processing and transportation resources, improves scheduling efficiency and safety, and reduces the risks of delays and conflicts.
Patent Information
- Application Number
- CN202611237759.0
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-14
- Publication Date
- 2026-11-17
AI Technical Summary
In flexible workshops with multiple overhead cranes, existing scheduling methods fail to effectively coordinate processing and transportation resources, resulting in a lack of coordination between processing and transportation plans. This can easily lead to machine waiting, workpiece delays, crane idle runs, and reduced equipment utilization. Furthermore, traditional methods have long computation times and insufficient strategy stability in complex scenarios.
A multi-task, multi-machine, and multi-crane collaborative scheduling method for flexible workshops based on hierarchical multi-agent deep reinforcement learning is adopted. Through hierarchical decision-making by agents for process-machine selection, crane allocation, and conflict avoidance, combined with Transformer encoders and cross-attention mechanisms, the joint scheduling of processing and transportation resources is optimized.
It improves the scheduling efficiency and transportation safety of flexible workshops, reduces workpiece delays and crane travel distances, enhances adaptive decision-making capabilities in complex production environments, and reduces the risk of conflicts between multiple cranes.
Smart Images

Figure CN123117613A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of intelligent manufacturing, industrial artificial intelligence, production scheduling optimization and material handling collaborative control, and in particular to a new method and system for dynamic monitoring of pollutants based on the coupling of multi-scale remote sensing data and environmental behavior mechanisms. Background Technology
[0002] In discrete manufacturing, large component processing, and heavy equipment manufacturing, workpieces typically undergo multiple processing steps sequentially according to a predetermined process route, with selection from multiple candidate machines based on the processing requirements of different steps. Due to the large size, weight, or long transport distance of the workpieces, material transfer between steps is usually accomplished by overhead conveying equipment such as bridge cranes, gantry cranes, or overhead cranes. Therefore, the machining process and the crane transport process are interconnected in time and space, and delays in any link can affect the execution of subsequent processes and the operational efficiency of the entire production system.
[0003] In workshops with a single overhead crane or relatively simple logistics relationships, the scheduling process typically considers factors such as workpiece processing sequence, machine allocation, and equipment load. However, in flexible work workshops with multiple overhead cranes, different cranes usually have their own service areas, and there may be overlapping areas or shared passageways between the work areas of adjacent cranes. When workpieces need to be transported across areas, task handover, buffering, and transportation sequence coordination between cranes are also involved. In addition, when multiple cranes perform transportation tasks simultaneously in adjacent areas, interference, blockage, or potential collisions may occur due to intersecting operating paths, overlapping operation times, or insufficient safety distances. Therefore, the scheduling of flexible work workshops with multiple overhead cranes not only needs to consider the matching relationship between processing tasks and machine resources, but also needs to comprehensively consider factors such as crane accessibility, transportation distance, empty-load operation, cross-area handover, conflict avoidance, and waiting costs.
[0004] In existing technologies, processing scheduling and material handling scheduling are typically carried out in stages or independently. Some methods first determine the processing sequence of workpieces and the machine allocation results, and then allocate transportation equipment based on fixed area rules, proximity principles, first-come-first-served rules, or manual experience. This type of method is relatively simple to implement and can generate feasible scheduling schemes in small-scale scenarios with fixed task relationships. However, because it fails to fully consider the coupling relationship between machine status, workpiece status, and crane status, it is prone to causing inconsistencies between processing plans and transportation plans. For example, the target machine may be idle but the workpiece has not yet been delivered, or the workpiece may have been processed but the available crane has not yet arrived, resulting in machine waiting, workpiece congestion, increased crane empty travel distance, and decreased equipment utilization.
[0005] For workshops where adjacent overhead crane service areas overlap, existing methods often employ preset priorities, fixed yielding rules, or manual intervention to handle crane conflicts. When the number of production tasks increases, transport routes frequently intersect, or multiple transport tasks enter the overlapping area within a similar timeframe, fixed rules struggle to adjust based on real-time task urgency, crane position, subsequent task load, and yielding costs. This can easily lead to congestion in shared areas, repeated waiting, frequent local conflicts, and extended overall completion time.
[0006] Traditional heuristic and metaheuristic algorithms can solve combinatorial optimization problems such as job sorting, machine allocation, and transportation task scheduling. However, as the number of jobs, candidate machines, and overhead cranes increases, the solution space expands rapidly. After changes in the workshop state, these methods typically require re-searching or iterating on a large number of candidate solutions, resulting in long computation times and difficulty in balancing solution quality and online scheduling efficiency.
[0007] In recent years, deep reinforcement learning has been applied to the workshop scheduling problem, learning scheduling strategies through interaction between agents and the production environment. Existing single-agent reinforcement learning methods typically represent process selection, machine allocation, crane selection, and conflict resolution as a unified joint decision problem. However, due to differences in constraints, time scales, and objects of action for different types of decisions, the state and action spaces can easily become excessively large, increasing the difficulty of feature representation, action selection, and reward allocation. Furthermore, a single global reward cannot accurately reflect the contribution of each decision-making stage to the final scheduling result, potentially leading to slow training convergence, insufficient policy stability, and weak generalization ability in complex scenarios. Summary of the Invention
[0008] To address the problems of insufficient coordination between processing and transportation resources, unreasonable allocation of cross-regional transportation tasks, and limited ability to coordinate conflicts among multiple overhead cranes in existing flexible workshops with multiple overhead cranes, this invention aims to provide a multi-task, multi-machine, and multi-overhead crane collaborative scheduling method for flexible workshops based on hierarchical multi-agent deep reinforcement learning. This method can achieve joint optimization of processing and transportation resources, reduce workpiece delays, crane travel distances, and the risk of conflicts among multiple overhead cranes, and improve the scheduling efficiency, transportation safety, and adaptive decision-making capabilities of flexible workshops in complex production environments.
[0009] To achieve the above-mentioned technical features, the objective of this invention is as follows: A flexible workshop multi-task, multi-machine, and multi-crane collaborative scheduling method based on hierarchical multi-agent deep reinforcement learning, comprising the following steps: S1. Obtain basic information on workpiece processes, candidate processing machines, overhead crane configuration, work areas and overlapping areas of adjacent areas in the flexible operation workshop, and establish a workpiece-machine-overhead crane collaborative scheduling model. S2. Based on the current workshop operating status, process sequence constraints, machine processing constraints, and overhead crane service area constraints, generate a candidate set of schedulable processes and machines for the current scheduling time. S3. Input the state features of the schedulable process-machine candidate set into the process-machine selection agent based on the Transformer encoder to determine the combination of the target process and the corresponding processing machine. S4. Generate material transportation tasks based on the combination of target process and corresponding processing machine. Input the candidate overhead crane status features, target workpiece features and target machine features into the overhead crane allocation agent based on cross-attention mechanism to determine the target overhead crane to perform the transportation task. S5. Based on the target crane's transportation start point, destination, estimated travel time window, and transportation route, detect the conflict relationship between the target crane and other cranes in the time and space dimensions. S6. When a spatiotemporal conflict is detected, the conflict association state is input into the crane conflict avoidance agent based on the Actor-Critic architecture, and the conflict coordination action of adjusting the yielding object, waiting time or transportation priority is output. S7. Execute joint scheduling actions that include process processing, material transportation and conflict avoidance, update the workshop operation status, update the network parameters of each agent according to the hierarchical reward and terminal reward, and repeat until all workpieces are processed. Output a collaborative scheduling scheme that includes process processing sequence, machine allocation, crane allocation, transportation path and waiting time.
[0010] Preferably, the flexible operation workshop is equipped with multiple processing machines, multiple workpieces to be processed, multiple overhead cranes, and multiple overhead crane service areas; overlapping areas or virtual handover points are set between adjacent overhead crane service areas, and cross-regional transportation tasks are completed through the overlapping areas or virtual handover points to complete the overhead crane inter-load handling.
[0011] Preferably, the workpiece-machine-crane collaborative scheduling model takes minimizing the maximum completion time, minimizing the total delay time, minimizing the total crane running distance, and minimizing the number of crane conflicts as joint optimization objectives, and is used to comprehensively evaluate processing efficiency, delivery time satisfaction, handling resource consumption, and transportation safety.
[0012] Preferably, the workpiece-machine-crane collaborative scheduling model satisfies the following constraints: Each process is processed by only one machine at any given time; each transport task is executed by only one overhead crane; each workpiece is moved to the finished product area after all processes are completed; subsequent processes are started after the preceding processes are completed and the corresponding transport tasks are completed; each machine processes at most one process at a time; each overhead crane executes at most one transport task at a time; cross-regional transport tasks meet the service range constraints of overlapping areas or handover points; when potential path conflicts exist, different overhead cranes occupy the same conflict space at different times through waiting, yielding, or priority adjustment.
[0013] Preferably, the status features in step S3 include the job completion rate, delay risk, current machine working status, machine utilization rate, cumulative machine waiting time, processing time of candidate processes on candidate machines, transportation time corresponding to candidate processes, target machine location, and current scheduling time.
[0014] Preferably, the process-machine selection agent based on the Transformer encoder sequentially includes an embedding layer, a self-attention encoding layer, and a policy output layer; the embedding layer maps the features of candidate process-machine combinations to a unified feature dimension, the self-attention encoding layer learns the global dependencies between candidate combinations, and the policy output layer outputs the action probability or matching score of each candidate combination.
[0015] Preferably, the crane allocation agent uses the candidate crane state vector as the query vector and the feature representation of the target transportation task as the key vector and value vector. It calculates the matching score between the candidate crane and the target transportation task through a cross-attention mechanism and determines the target crane based on the matching score.
[0016] Preferably, the candidate overhead crane status characteristics include the current overhead crane position, overhead crane availability time, overhead crane load status, distance from the overhead crane to the workpiece position, distance from the overhead crane to the target machine, cumulative transportation distance, service area, and potential conflict relationships with other overhead cranes.
[0017] Preferably, when the starting and target locations of a transportation task are within the same overhead crane service area, the overhead crane to be executed is directly determined according to predefined area rules; when the transportation task involves cross-regional handling or there are multiple feasible candidate overhead cranes, the overhead crane allocation agent based on the cross-attention mechanism is invoked to make a decision.
[0018] Preferably, the conflict association status in step S6 includes the degree of overlap in the time of adjacent overhead crane tasks, the degree of overlap in the spatial path, the current number of conflicts in the workshop, whether each overhead crane has any tasks to be executed, the current position of the conflicting overhead crane, the target position, the expected waiting cost, and the expected transportation impact.
[0019] Preferably, the transport priority adjustment action includes selecting the current overhead crane to give way, selecting the conflicting overhead crane to give way, delaying the current transport task, delaying the conflicting transport task, and waiting at the overlapping area or virtual handover point.
[0020] Preferably, the tiered rewards include process-machine selection rewards, overhead crane allocation rewards, and conflict avoidance rewards; The process-machine selection bonus is associated with reduced delay risk, improved completion time, and increased scheduling progress. The overhead crane allocation reward is related to reduced transport distance, changes in potential conflicts, and improved transport efficiency; The conflict avoidance rewards are related to conflict resolution, yielding costs, changes in transport distance, changes in delays, conflict increments, and penalties for illegal actions.
[0021] Preferably, the terminal reward is calculated based on the final maximum completion time, total delay time, cumulative crane running distance, number of completed processes, and number of unresolved conflicts, and is adaptively normalized according to the scale of the current scheduling scenario.
[0022] Preferably, the network parameters of each agent are trained through policy gradient, advantage function, empirical sampling or target network update, so that agents at each layer can cooperate between local scheduling benefits and global scheduling objectives.
[0023] The present invention has the following beneficial effects: 1. This invention models process-machine selection, overhead crane allocation, and conflict avoidance as hierarchical intelligent agent decisions, reducing the learning difficulty of a single intelligent agent in a high-dimensional combined action space and improving the trainability and interpretability of complex workshop scheduling problems.
[0024] 2. This invention utilizes a Transformer encoder to perform global correlation modeling of candidate process-machine combinations, enabling scheduling decisions to simultaneously consider factors such as job progress, delay risk, machine status, and processing time, thereby improving the rationality of process-machine matching.
[0025] 3. This invention utilizes a cross-attention mechanism to characterize the matching relationship between candidate overhead cranes and target transportation tasks, and can dynamically select the crane to be executed based on the crane's location, available time, running distance, and regional constraints, thereby reducing empty runs and potential conflicts.
[0026] 4. This invention employs an Actor-Critic structure for multi-crane conflict avoidance, selecting the yielding object based on local conflict status and global value assessment, so that the passage order of multiple cranes in overlapping areas can be dynamically adjusted according to operating costs and scheduling impact.
[0027] 5. This invention guides learning through hierarchical rewards and terminal rewards, ensuring that local action optimization aligns with the global scheduling objective, thereby improving solution set convergence and overall scheduling performance under different scales of computational examples. Attached Figure Description
[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.
[0029] Figure 1 A schematic diagram of a multi-task, multi-machine, multi-crane layout for a flexible operations workshop; Figure 2 A Gantt chart example for small-scale, multi-day train scheduling; Figure 3 This is a diagram of the hierarchical multi-agent deep reinforcement learning scheduling framework of the present invention; Figure 4 This is a diagram showing the analysis results of the Taguchi test for key parameters of this invention; Figure 5 A comparison chart of the average rank of different algorithms; Figure 6 The average reward iteration curves of different single-agent algorithms in multi-scale tests are shown. Figure 7 This is a Pareto solution set distribution diagram of the algorithm of this invention and the comparative multi-agent algorithm. Detailed Implementation
[0030] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.
[0031] Example 1: This embodiment provides a multi-task, multi-machine, and multi-crane collaborative scheduling scenario for a flexible workshop. The workshop includes m machines, n workpieces, r cranes, and a processing areas, with some overlap between adjacent processing areas. Each workpiece includes multiple sequentially executed processes, and each process has a set of candidate machines. The processing time for different candidate machines can be different. Each workpiece has an ideal delivery date; if the workpiece completion time exceeds the ideal delivery date, a delay occurs.
[0032] The issues explored in this study are based on the following assumptions: (1) The preparation time before crane handling and machine processing is zero.
[0033] (2) The first processing step of each workpiece must be transported from the raw material area to the target machine by the overhead crane, and after all processing steps are completed, the workpiece is transported to the finished product area by the overhead crane.
[0034] (3) The overhead crane operates at a constant speed during transportation.
[0035] (4) During the transportation process, the overhead crane does not consider the obstruction of obstacles, and the transportation distance is determined by the straight-line distance between the starting point and the ending point.
[0036] (5) The workshop is divided into multiple areas, and each area is equipped with an overhead crane.
[0037] (6) Except for cross-regional tasks, each crane can only perform transportation tasks in the area it is responsible for.
[0038] (7) The overhead crane can only move one workpiece at a time, and the workpiece cannot be put down midway during transportation.
[0039] (8) The machine can only process one workpiece at a time, and the processing process cannot be interrupted.
[0040] In this embodiment, the overhead crane is responsible for three types of transport tasks: the first type is the transport of workpieces from the raw material area to the target machine of the first process; the second type is the inter-process transfer of workpieces between adjacent or non-adjacent machines; and the third type is the transport of workpieces from the machine of the last process to the finished product area after all processes are completed. When the transport task involves cross-regional transport, the workpieces can be relayed within the overlapping area of adjacent overhead crane service areas or at virtual handover points.
[0041] Example 2: The workpiece-machine tool-overhead crane collaborative scheduling problem in the flexible manufacturing workshop studied in this invention can be defined as follows: The workshop includes m machine n One workpiece, r Sky crane and a Each processing area has a certain degree of overlap with other processing areas. Each workpiece... i Includes Each process involves multiple machines to choose from, and different workpieces can be processed simultaneously. i different machines k Processing time There are differences; each workpiece i There is an ideal delivery time. Each overhead crane is responsible for transporting all workpieces in its assigned area, specifically including: the cranes working together to move workpieces from the raw material area (referred to as virtual machine 0) to the first processing machine, inter-process machine transfers, and from the last processing machine to the finished product area (referred to as virtual machine 0). m +1), including the overhead crane r From the machine Move to machine The required time is (If two overhead cranes collide while performing tasks in an overlapping area, a collision avoidance agent will coordinate the process.) As the processing status changes continuously, the flexible workshop needs to make joint decisions on machine-workpiece matching, crane coordination, and collision avoidance between cranes based on the current workpiece and machine status.
[0042] In summary, the problems studied in this invention mainly include the following three sub-problems: (1) Machine-workpiece matching problem, that is, according to the current machine status and workpiece processing requirements, select appropriate processing equipment for the process to be processed from the set of available machines to achieve reasonable matching between the process and the machine; (2) Overhead crane selection problem, that is, under the premise of meeting transportation time and accessibility, select appropriate overhead cranes to perform transportation tasks for the workpiece to be transported to achieve effective allocation of overhead crane resources; (3) Overhead crane collision avoidance problem, that is, when multiple overhead cranes are running at the same time, avoid spatial conflicts or blockages between overhead cranes by reasonably planning the running sequence and movement path of the overhead cranes to ensure the safety and efficiency of the transportation process.
[0043] In this study, a multi-objective optimization model was established to simultaneously minimize the total project duration, total delay time, crane travel distance, and number of crane collisions. The mathematical expression of the model is as follows: (1) (2) (3) (4) constraint: (5) (6) (7) (8) (9) (10) (11) (12) (13) (14) (15) (16) (17) Constraint (5) indicates that each process can only be processed on one machine at any given time; Constraint (6) indicates that a transport task can only be completed by one overhead crane; Constraint (7) indicates that each workpiece needs to be transported to the finished product area after completing all processes; Constraint (8) indicates that the workpiece i The next process can only begin after the previous process is completed; constraints (9)-(10) indicate that a machine can only process one process at a time; constraint (11) the workpiece can only begin processing after the overhead crane has completed the handling of the workpiece; constraint (12) is used to calculate the workpiece i The j This process is done on the machine. k The processing end time on the crane; constraint (13) is used to calculate the crane's processing end time. r Load handling workpiece i From the machine To the machine k The end time; constraint (14) is used to calculate the overhead crane. r Load handling workpiece i From the machine k To the machine The end time; constraint (15) is used to calculate the overhead crane. r Load handling workpiece i From the machine k To the machine The start time; constraint (16) indicates the overhead crane r The next workpiece handling task can only be performed after the previous handling task has been completed.
[0044] To facilitate understanding of the problem studied in this invention, an illustrative example is used: Assume there are 4 workpieces in the workshop, each workpiece has 4 processes, and the processing time of all processes is 10 or 15; and the workshop is divided into two areas, each area is equipped with one overhead crane and 3 machines, and the transportation time of all overhead cranes is 2 or 3.
[0045] To facilitate patent implementation, the multi-objective optimization model can be expressed as minimizing F1, F2, and F3, where F1 represents the maximum completion time, F2 represents the total delay time for all workpieces, and F3 represents the total travel distance of all cranes. The above objectives can be comprehensively optimized using weighted summation, hierarchical ranking, Pareto dominance, or reinforcement learning reward functions.
[0046] In this embodiment, the constraints include: process sequence constraints, machine-specific processing constraints, crane-specific handling constraints, processing after transportation constraints, crane service area constraints, cross-regional handover constraints, overlapping area capacity constraints, and multi-crane conflict avoidance constraints. These constraints ensure that the output scheduling scheme satisfies both process feasibility and the operational safety of multiple cranes.
[0047] Symbol definition
[0048] Example 3: To address the integrated scheduling problem of task selection, resource allocation, and conflict resolution in complex manufacturing systems, this invention proposes a multi-stage reinforcement learning scheduling framework, such as... Figure 3 As shown, the framework consists of three decision-making agents: a task selection agent, a crane allocation agent, and a multi-crane conflict resolution agent. At each decision-making moment, the shop floor environment first obtains the currently schedulable job-machine combinations and constructs state features. These features include shared features describing the overall shop floor state and resource conditions, as well as private features related to candidate jobs, target machines, and candidate cranes. The task selection agent uses a Transformer encoder to evaluate all schedulable job-machine combinations and selects the next processing step. Subsequently, the crane allocation agent uses a cross-attention mechanism to match the selected task with a suitable crane.
[0049] When a potential spatiotemporal conflict is detected between overhead cranes, the conflict resolution agent will make cooperative obstacle avoidance decisions, such as waiting, buffering, or reallocating. After the action is executed, the environment updates the scheduling state and calculates the reward value. The reward design comprehensively considers scheduling efficiency and operational safety, including maximum completion time, delay, transportation distance, scheduling progress, overhead crane conflict penalties, and additional rewards or penalties during the conflict resolution process. This process is repeated continuously until all processes are completed, and the policy network is optimized through policy gradient learning.
[0050] Specifically, the hierarchical multi-agent reinforcement learning framework breaks down a complete scheduling decision into three consecutive but mutually reinforcing stages. In the first stage, the process-machine selection agent processes a set of candidate tasks to determine the current processing task to be executed. In the second stage, the crane allocation agent determines the crane to be executed based on the target transport task and the status of candidate cranes. The third stage is initiated when a potential conflict is detected, where a conflict avoidance agent determines the yielding object or the waiting action. The decision results of the three stages collectively affect the shop floor environment, and the rewards from the environmental feedback are used for both local policy updates and global performance optimization.
[0051] Example 4: In multi-day machine shop scheduling, task selection directly determines subsequent resource allocation and system operating efficiency. Therefore, this invention constructs a task selection agent based on a Transformer encoder to dynamically select the optimal task from candidate workpiece-machine combinations. At any scheduling time The workshop environment generates a set of schedulable tasks based on the current processing status and constraints. For each candidate task, a multi-dimensional state feature vector is extracted from the environment. This feature comprehensively characterizes information such as workpiece processing progress, delay level, delivery urgency, machine load level, waiting time, and transportation time. All candidate task features are stacked sequentially to form an input matrix. And by embedding it into a high-dimensional representation space through linear mapping, we obtain ,in These are learnable parameters. Subsequently, the embedded sequence is input into the Transformer encoder, which models the global dependencies between candidate tasks through a multi-layer self-attention mechanism. This ensures that each task representation not only depends on its own features but also incorporates contextual information from other tasks, resulting in the encoded output. To further incorporate the influence of the overall system state on task priority, this invention employs a global feature aggregation method to construct the system state vector. And mapped to a query vector through a linear transformation. v. Based on this, the priority score of each candidate task is calculated using a dot product attention mechanism. The task selection probability distribution is obtained by normalizing it using the Softmax function, and finally the optimal task at the current moment is obtained.
[0052] To comprehensively characterize the priority of each workpiece task in the workshop, the common features of this intelligent agent are designed as follows: (1) Percentage of processes completed for the current workpiece (18) (2) Risk of delay for current workpiece (19) in, To prevent division by zero, Indicates the current scheduling time of the workshop. Indicates the number of optional machines for a process.
[0053] (3) Current machine busy status (20) (4) Current machine utilization rate (twenty one) (5) Current machine waiting time (twenty two) The private characteristics of this agent are shown in the table below:
[0054] During the task selection phase, the action space is defined as the set of all executable artifact-machine combinations at the current scheduling moment. At the scheduling moment... The workshop environment generates a set of candidate tasks, therefore the action space of the agent can be represented as a discrete set, where each action corresponds to selecting the first... A number of candidate tasks are executed. These are further mapped to a probability distribution using the Softmax function. During the decision-making process, the agent samples according to this probability distribution to obtain the final action, thus achieving dynamic selection of candidate tasks. This action space is characterized by dynamic changes with the environmental state, and its size is determined by the number of currently schedulable tasks, enabling the model to flexibly adapt to the decision-making needs of different scheduling stages.
[0055] The task selection agent is responsible for choosing suitable workpieces and processing machines from the current set of schedulable tasks. Its decisions directly affect the subsequent processing sequence, machine load distribution, and workpiece delay risk. Therefore, the reward at this layer primarily focuses on changes in scheduling progress, completion time, and delay risk. Positive feedback should be given if a decision can advance more processes, reduce expected delays, or shorten the overall completion time; conversely, penalties should be imposed if the decision leads to increased completion time or increased delay risk. In implementation, the process-machine selection agent and the crane allocation agent work together on a complete scheduling decision, thus sharing global scheduling feedback. For ease of explanation, the reward items related to process-machine selection are represented as follows: (twenty three) in, This indicates that the expected improvement bonus will be postponed. Indicates an improvement bonus for completion time. This represents the scheduling progress reward. This reward guides the agent to prioritize actions that reduce delay risks, advance process completion, and improve overall scheduling efficiency. These metrics are calculated based on changes in scheduling metrics before and after production-equipment decisions: (twenty four) The relevant variables are defined as follows: (25) (26) (27) The normalization factor is defined as: (28) (29) (30) The relevant variables are defined as follows: (31) In this embodiment, the Transformer encoder may include an input embedding layer, a multi-head self-attention layer, a feedforward network layer, and a policy output layer. The feature vectors of candidate task-machine combinations are first linearly mapped to obtain an embedding representation. Then, the competitive and cooperative relationships between candidate combinations are learned through a multi-head self-attention mechanism. Finally, the action probability of each candidate combination is obtained through the policy output layer. Since the number of candidate tasks varies with scheduling time, the agent can mask inactive actions through a masking mechanism.
[0056] Example 5: After task selection, it is necessary to further determine the optimal overhead crane to execute the task in order to achieve efficient resource allocation. To this end, this invention constructs an overhead crane allocation agent based on a cross-attention mechanism to characterize the matching relationship between overhead cranes and tasks. During scheduling... Given the selected task and the workshop environment, provide the state set of all overhead cranes. For each overhead crane... The system extracts the feature vector of the crane's state and, combined with the workpiece and machine features corresponding to the task, constructs a task representation vector h. Through linear mapping, the crane's state and task features are embedded into a unified feature space. Using the crane as the query and the task as the key and value, a cross-attention mechanism is built to calculate the matching degree between the crane and the task. The core calculation process can be represented as follows: (32) in Indicates the first The query vector for the overhead crane. The key vector represents the task. It is further normalized at the crane dimension using the Softmax function to obtain the crane allocation probability. This probability reflects the suitability of each crane for executing the current task. Finally, the optimal crane is selected to execute the transportation task based on this distribution, and the decision result is passed to the subsequent obstacle avoidance module. In this way, the model can comprehensively consider crane position, task attributes, and system state from a global perspective, achieving dynamic modeling of the "crane-task" matching relationship. During training, this allocation strategy is embedded in a reinforcement learning framework, and the parameters are optimized through the reward signal of the scheduling result, thereby achieving adaptive optimization of the crane allocation strategy under multiple objective constraints such as transportation cost, resource utilization, and system load balancing.
[0057] To accurately depict the matching relationship between the overhead crane and the task, this invention constructs a multi-source feature representation for the intelligent agent that integrates the crane state, workpiece state, and machine state. Based on the shared features, private features that reflect individual differences and local decision-making information are further designed, the specific structure of which is shown in the table below.
[0058]
[0059] Specifically: Features (1) and (2) belong to the task urgency feature, where the theoretical transportation time reflects the basic time cost required to complete the current handling task, while the delivery period describes the time constraint of the task and its urgency; Features (3), (5) and (6) belong to the spatial location feature, where the spatial location of the machine is used to characterize the distribution of tasks in the workshop layout, and the distance from each crane to the workpiece location and the target machine location corresponds to the empty travel cost and the loaded transportation cost of the crane, respectively; Feature (4) belongs to the resource status feature, which describes the availability level of transportation resources in the system by characterizing the busy status of each crane.
[0060] During the crane allocation phase, the agent's task is to assign a suitable crane to the selected workpiece-machine combination to perform the transportation task. Therefore, its action space is defined as the set of all available cranes in the current system. At the scheduling time... Let the set of overhead cranes be... If ; then the action space of the agent can be represented as a discrete set. .in Indicates the selection of the first The overhead crane is performing its current task.
[0061] Based on the cross-attention mechanism, the policy network first calculates the matching score between each crane and the task, and then normalizes it along the crane dimension using the Softmax function to obtain the crane allocation probability distribution. It should be noted that in actual scheduling, if the starting and destination positions of the task are both within the same service area, the crane selection is directly determined by predefined rules; the agent is only invoked to make decisions when the task involves cross-regional transportation. Therefore, while the action space is structurally a fixed-dimensional discrete space, it is influenced by regional constraints during actual execution, thus exhibiting certain conditional decision-making characteristics.
[0062] After the process-machine combination is determined, the crane allocation agent further selects the crane to perform the transportation task. This decision not only affects the transportation time of the current workpiece but also the spatial occupancy relationship between cranes and the potential conflict risk. Therefore, the reward at this layer mainly focuses on the transportation distance, conflict increment, and impact on the overall completion time. A reasonable crane allocation should minimize the empty and loaded transportation distances, reduce the possibility of spatiotemporal conflicts with other cranes, and avoid additional waiting or increased completion time due to improper crane selection. Similar to the first layer, the crane allocation agent shares global scheduling rewards during training. This invention represents the reward items related to crane allocation as follows: (33) in, This indicates an incentive for improving transportation distance. Indicates potential incremental penalties for conflict. This represents the overall efficiency between transportation and scheduling. The reward incentivizes agents to select overhead cranes with shorter distances, lower conflict risks, and greater overall scheduling performance. These metrics are calculated based on changes in transportation metrics before and after the overhead crane allocation decision: (34) The relevant variables are defined as follows: (35) (36) (37) In this embodiment, if the starting location of the target process and the target machine are both located within the same overhead crane service area, the system can directly select the overhead crane in that area to perform the transportation task according to the area rules. If the target process involves cross-regional transportation, or is located in an overlapping area covered by multiple overhead cranes, then the cross-attention overhead crane allocation agent is invoked. This design can reduce unnecessary agent invocations and maintain decision-making flexibility in complex cross-regional tasks.
[0063] Example 6: After crane allocation, if multiple cranes have overlapping paths and time windows during transportation, further conflict coordination decisions are needed to ensure the safety and continuity of system operation. To this end, this invention constructs a crane collision avoidance decision model based on a multi-agent Actor-Critic framework, treating each crane as an independent agent and enabling collaborative decision-making in potential conflict scenarios. During scheduling... , No. The overhead crane extracts local observation vectors based on current task information and workshop status: (38) in, , , ,and These represent workpiece characteristics, target machine characteristics, crane's own state characteristics, and system global characteristics, respectively. The Actor network uses local observation... Given the input, output the policy distribution in the current state: (39) in, This represents a policy network composed of multiple sensing layers. Indicates the first Coordination actions taken by the overhead crane in a conflict situation.
[0064] To enhance the collaborative capabilities of multiple overhead cranes, this invention employs a centralized Critic system to perform a global value assessment of the joint state of all cranes. The joint state is obtained by stitching together the local observations of each crane. And estimate state value through a value network. During training, an immediate reward is constructed based on the state transition after conflict resolution, and the advantage function is calculated: (40) This advantage function is used to guide the Actor network update, enabling the collision avoidance agent to learn effective cooperative decision-making strategies under different conflict states.
[0065] For the collision avoidance decision-making agent of overhead cranes, conflict-related private features are introduced in addition to public features to characterize the spatiotemporal interaction relationship between overhead cranes, thereby improving the overall scheduling efficiency of the production workshop. Its specific composition is shown in the table below.
[0066]
[0067] Among them, the time overlap of adjacent overhead crane tasks is used to measure whether there is a time overlap in the execution of tasks by different overhead cranes. This indicator can reflect the time urgency of potential conflicts. The spatial overlap of adjacent overhead crane tasks is used to characterize the spatial intersection of overhead crane operation paths. This feature directly reflects the spatial conditions for the occurrence of conflicts. The number of conflicts in the workshop is used to describe the overall congestion and conflict pressure of the current system. Whether there are pending tasks for each overhead crane is used to reflect the utilization status of system resources and task distribution.
[0068] In the multi-agent conflict coordination model constructed in this invention, each overhead crane is considered an independent agent. Unlike directly outputting low-level motion actions such as "forward / wait," this invention models the conflict coordination process as an avoidance agent selection problem. When the system detects a potential spatiotemporal conflict between the current transport task and other cranes, it first constructs a conflict participant set Ω, which consists of the crane currently performing the transport task and the cranes with which it has a potential conflict. Subsequently, the conflict coordination agent evaluates the candidate cranes in the conflict participant set based on each crane's local state, task state, expected avoidance cost, and conflict relationship, and selects one as the avoidance target. Therefore, the action space of the conflict coordination agent can be defined as... .
[0069] in, This indicates the crane number selected as the obstacle to be avoided. If the crane currently performing the task is selected as the obstacle to be avoided, the current transport task is postponed; if another conflicting crane is selected as the obstacle to be avoided, the most recent transport task of that crane is postponed, allowing the current transport task to gain priority. In this way, the model can learn priority relationships based on the operating status and avoidance costs of different cranes when a potential conflict is detected, thereby achieving dynamic coordination among multiple cranes.
[0070] To guide the conflict coordination agent in selecting appropriate avoidance targets, this invention designs a local conflict coordination reward function. This reward function not only considers whether the conflict is successfully resolved, but also factors such as avoidance costs, scheduling delays, changes in transport distance, and illegal avoidance choices. Its aim is to enable the agent to select overhead cranes with lower costs and greater benefits to overall scheduling performance in conflict scenarios. The local reward function is defined as follows: (41) in This indicates the consequences and penalties for avoiding the situation; Indicates a reward for the progress of the scheduling; Indicates a reward for changes in transportation distance; Indicates incremental penalty for conflict; Indicates a penalty for incremental completion time; This indicates the penalty for an illegal action. The calculation method is as follows: (42) Through the above reward design, the conflict coordination agent can not only learn how to resolve potential conflicts, but also weigh the waiting costs, scheduling delays and transportation impacts among multiple avoidable objects, thereby choosing a more reasonable way to avoid the overhead crane.
[0071] In this embodiment, the conflict participant set consists of the overhead crane currently performing a transport task and other overhead cranes that may potentially conflict with it. If the current overhead crane is selected as the yielding object, the current transport task is postponed; if another conflicting overhead crane is selected as the yielding object, the most recent transport task of that conflicting overhead crane is postponed, allowing the current transport task to gain priority. In this way, the model can learn dynamic priority relationships based on the operating status of different overhead cranes, the urgency of the task, and the yielding cost.
[0072] Example 7: In addition to the step-by-step rewards mentioned above, this invention introduces a termination reward at the end of each scheduling round to evaluate the final quality of the complete scheduling scheme. The termination reward measures the final completion time, total delay, cumulative transportation distance, completion progress, and number of unresolved conflicts from a global perspective, enabling the agent to not only focus on the gains of single-step decisions but also learn long-term scheduling goals. The termination reward is defined as: (43) in, and These represent the final completion time, total delay, cumulative transportation distance, number of completed processes, and number of unresolved conflicts, respectively. and An adaptive normalization scale is constructed based on the current sample size.
[0073] Through adaptive normalization, the reward terms for different scale examples are transformed into a relatively stable numerical range, thereby avoiding large-scale examples from dominating the training process due to large completion time, transportation distance or number of processes.
[0074] In this embodiment, the process-machine selection reward, the crane allocation reward, and the conflict avoidance reward together constitute a hierarchical reward system. The process-machine selection reward encourages the agent to prioritize tasks that can reduce the risk of delays, advance the process completion, and improve the completion time; the crane allocation reward encourages the agent to select cranes with short distances, high availability, and low conflict risk; the conflict avoidance reward encourages the agent to resolve potential conflicts with minimal yielding costs and penalizes illegal actions, increased conflicts, and deterioration of completion time.
[0075] Example 8: To verify the effectiveness of the proposed algorithm and strategy, 27 test cases were randomly generated (denoted as: scale-random seed). The case generation rules are as follows: (1) All machines are arranged linearly along the axis in numerical order, with the raw material area at the leftmost end and the finished product area at the rightmost end. The basic spacing between the actual processing machines is 40, and random perturbation is added in the range of [-3,3]. (2) The workshop is divided into crane operation areas according to the machine position. A certain overlap range is set between adjacent areas, while ensuring that adjacent cranes are covered by real machines near the boundary to meet the cross-regional handover requirements. (3) For cross-regional transportation tasks, the workpiece needs to complete the relay at the virtual handover point in the overlapping area of adjacent cranes. The handover point is set with a limited buffer capacity to simulate temporary waiting in actual production. (4) The processing time on each candidate machine is randomly generated in the range of [750,2200]. The processing time of non-candidate machines is recorded as -1, and the processing time of the virtual process corresponding to the raw material area and the finished product area is 0.
[0076] In addition, all programs were implemented in Python and independently trained 500 times and tested 100 times on a desktop computer with a Windows 11 environment, an AMD EPYC750232-Core Processor, and 512GB of RAM. The algorithm performance was evaluated based on the test results.
[0077] This patent selects two classic evaluation metrics widely used in the field of multi-objective optimization—Inverted Generational Distance (IGD) and Hypervolume (HV)—to measure algorithm performance. Among them, the IGD metric is mainly used to measure the convergence of the solution set, while the HV metric can simultaneously characterize the convergence, diversity, and distribution of the solution set.
[0078] (1) IGD is the average distance from each point in the Pareto front to the nearest solution in the algorithm's solution set (to avoid the influence of different target scales on the distance calculation, the values of each objective function are normalized before calculating IGD), and the specific calculation formula is as follows: (44) Among them, Pareto frontier P The solution set is composed of the non-dominated solutions obtained by all the comparison algorithms, where A is the solution set obtained by a certain algorithm. It is worth noting that the smaller the IGD, the better the convergence of the solution set.
[0079] (2) The HV is the hypervolume bounded by the normalized Pareto front and a reference point of 1.1 times, and the specific calculation formula is as follows: (45) Where Vol(·) is the Lebesgue measure, and fng is the normalized objective function value of the g-th objective (g=1,2,3). It is worth noting that a larger HV indicates better overall population performance.
[0080] To facilitate comparison of the differences between different methods or strategies, this invention uses relative percentage aged evidence (RPD) to analyze the evaluation index, and the specific calculation process is as follows: (46) Wherein, Iother is the evaluation metric for other algorithms, and Iour is the evaluation metric for the algorithm of this invention. In this embodiment, the IGD index is used to measure the average distance between the solution set obtained by the algorithm and the reference Pareto front. The smaller the value, the closer the solution set is to the reference front. The HV index is used to measure the hypervolume enclosed by the normalized solution set and the reference point. The larger the value, the better the overall performance of the solution set in terms of convergence, diversity and distribution. The RPD index is used to perform statistical analysis on the relative deviation of different algorithms on the same evaluation index.
[0081] Example 9: I. Parameter Settings In the TCAC-HMARL algorithm, the learning rate... Embedded Dimension Number of encoder stacking layers Discount Factor and the number of attention heads All of these parameters significantly impact algorithm performance. To analyze their influence, a Taguchi experimental design method was employed, with three different levels for each parameter: η∈{0.0001,0.0003,0.0001}, γ∈{0.95,0.99,0.995}, L∈{1,2,3}, D∈{32,64,128}, and H∈{2,4,8}. Based on this, 27 orthogonal parameter combinations were constructed, and each combination was independently tested 500 times on a medium-scale computational example.
[0082] To facilitate the evaluation of response effects under different parameter settings, the HV index was selected as the response value in the orthogonal analysis. Specific analysis results are as follows: Figure 4 As shown. Based on the signal-to-noise ratio principle of "the higher the better", the optimal parameter combination is determined as follows: , , , and The importance of the above key parameters is ranked as 3, 5, 1, 2, and 4, respectively.
[0083] II. Effectiveness Analysis of the TCAC-HMARL Algorithm (I) Effectiveness Analysis of Hierarchical Multi-Agent Cooperative Mechanism To verify the effectiveness of the hierarchical multi-agent cooperative mechanism in the TCAC-HMARL algorithm proposed in this invention, this section removes the first-layer task selection agent, the second-layer crane allocation agent, and the third-layer conflict coordination agent from the TCAC-HMARL algorithm, forming three ablation algorithms: w / oTaskAgent, w / oCraneAgent, and w / oCollisionAgent. Running all test examples revealed that: (1) the Wilcoxon test p-values of the TCAC-HMARL algorithm and the three ablation algorithms at 95% confidence levels were 5.80×10⁻³, 5.80×10⁻³, and 2.76×10⁻², respectively, indicating significant differences between the TCAC-HMARL algorithm and each ablation algorithm; (2) the average IGD and HV of TCAC-HMARL were 0.5341 and 0.6773, respectively, which are generally superior to the three ablation algorithms. Based on TCAC-HMARL, the HV metrics RPD of TCAC-HMARL-TaskAgent, TCAC-HMARL-CraneAgent, and TCAC-HMARL-CollisionAgent are as follows: 77.04%, 75.64% and The performance degradation rates were 67.90%, and the RPD values for the IGD index were 96.29%, 63.68%, and 59.15%, respectively. This indicates that removing any layer of agents leads to a decrease in algorithm performance, with the most significant performance degradation occurring when removing the agent responsible for task selection.
[0084] (II) Comparative Analysis with Single-Agent Algorithms To further verify the advantages of the TCAC-HMARL algorithm in solving complex shop floor scheduling problems, this invention selects four typical single-agent reinforcement learning algorithms—DQN, D3QN, PPO, and AC—as comparison algorithms and conducts comparative experiments on all test cases. Experimental results show that: (1) The average IGD and HV of TCAC-HMARL are 0.5216 and 0.6389, respectively, which are generally superior to the four single-agent algorithms. Based on TCAC-HMARL, the HV index RPD of DQN, D3QN, PPO, and AC are respectively... 56.28%, 52.56%, 69.09% and 76.10%, and the IGD index RPD were 71.43%, 40.11%, 73.18%, and 81.02%, respectively. It can be seen that the HV of the four single-agent algorithms all decreased significantly, and the IGD all increased significantly; (2) The average rank of TCAC-HMARL was 3.74, which was higher than that of DQN, D3QN, PPO and AC (3.13, 2.98, 2.70 and 2.56), indicating that TCAC-HMARL has better overall performance and stability; (3) The Wilcoxon test P values of TCAC-HMARL algorithm and the other four typical single-agent algorithms on HV were 2.26×10. - ¹、8.08×10 - ²、2.17×10 - ² and 1.38×10 - ². The p-values between TCAC-HMARL and PPO and AC are all less than 0.05, indicating that there are significant differences between TCAC-HMARL and PPO and AC. Although the differences between TCAC-HMARL and DQN and D3QN did not reach the significance level of 0.05, TCAC-HMARL still showed better average index values and overall ranking; (4) Figure 6The average cumulative reward curves and 95% confidence intervals for each algorithm are presented under small, medium, and large-scale examples. It can be seen that TCAC-HMARL achieves higher cumulative rewards across all three scales, especially in medium and large-scale examples where it shows a stable upward trend with increasing scheduling steps. In contrast, DQN, D3QN, PPO, and AC have generally lower cumulative rewards, with most curves showing a decline in the later stages, indicating that single-agent algorithms struggle to effectively handle collaborative decisions such as task selection, crane allocation, and conflict resolution. Overall, TCAC-HMARL demonstrates better stability and adaptability, making it more suitable for multi-resource collaborative workshop scheduling problems.
[0085] (III) Comparative Analysis with Multi-Agent Algorithms To verify the superiority of the algorithm of this invention over common multi-agent reinforcement learning algorithms, MADDPG and BiCNet were selected as comparison algorithms. Running all test cases revealed that: (1) TCAC-HMARL's average IGD and HV were 0.4157 and 0.7960, respectively, which are superior to MADDPG and BiCNet. Based on TCAC-HMARL, the HV index RPD of MADDPG and BiCNet were respectively... 88.63% and 85.16%, and the IGD index RPD were 163.46% and 162.21%, respectively. This indicates that compared with TCAC-HMARL, the HV of the two multi-agent comparison algorithms is significantly reduced and the IGD is significantly increased; (2) Under the 95% confidence condition, the Wilcoxon test P values of TCAC-HMARL and the four multi-agent algorithms on the HV value are 5.25×10-5 and 8.17×10-6, respectively, indicating that there are significant differences between the TCAC-HMARL algorithm and the two multi-agent algorithms; (3) By analyzing the solution set distribution diagram of the three multi-agent algorithms, it can be seen that (e.g. Figure 7 As shown in the figure, the TCAC-HMARL algorithm has a better solution set than other multi-agent algorithms.
[0086] III. Sensitivity Analysis (I) Overlap Coefficient of Overhead Crane Area Sensitivity analysis To analyze the impact of the overlap coefficient of the overhead crane area on the production scheduling results, this invention sets it to three different values, namely... Running all test cases revealed that the time-comprehensive performance is optimal for small-scale problems; for medium-scale problems, ... Superior to the other two parameter values; in large-scale problems, Both 0.2 and 0.2 perform well, but in terms of overall rank and stability, It is even more advantageous. Further analysis of HV and IGD metrics across all scales reveals that... The average HV is higher and the average IGD is lowest, resulting in the best overall ranking; while when At this point, the algorithm performance significantly degrades. This indicates that excessive overlap in the crane area increases the coordination complexity and potential conflicts during the scheduling process, hindering the acquisition of a stable scheduling solution. Therefore, the experiments in this invention set... .
[0087] (II) Delivery Time Tension Factor Sensitivity analysis To analyze the impact of the delivery time tension coefficient 'a' on scheduling performance, this invention sets it to three different values, namely... Three levels. Experimental results show that in small- and medium-scale examples, Best performance; in large-scale cases, Better results were achieved for HV and IGD. Overall, the statistical results show... The average HV was the highest, the average IGD was the lowest, and the overall rank was optimal. This indicates that reasonably increasing the delivery time tension coefficient helps the algorithm achieve a better balance between completion time, total delay, and transportation distance. Therefore, the experiment in this invention sets... .
[0088] The experimental results show that the complete hierarchical structure has significant advantages over the ablation model that removes any layer of agents, indicating that the three decision layers—task selection, crane allocation, and conflict coordination—all contribute to the final scheduling performance. Compared to single-agent algorithms, this method avoids the single-policy network bearing an excessively large combined action space; compared to ordinary multi-agent algorithms, this method improves training stability and collaborative optimization capabilities through decision-level partitioning and reward decomposition.
[0089] Furthermore, sensitivity analysis shows that an excessively large overlap coefficient in the overhead crane area increases coordination complexity and potential conflicts, which is detrimental to obtaining a stable scheduling solution. Appropriately increasing the delivery time pressure coefficient helps the algorithm achieve a better balance between maximum completion time, total delay time, and transportation distance. Therefore, in engineering applications, the area overlap range and delivery time pressure coefficient can be jointly calibrated based on workshop layout density, overhead crane service radius, and delivery time pressure.
[0090] Example 10: This invention can also be applied to workshop digital twin systems, manufacturing execution systems, or intelligent scheduling systems. First, basic data is read from production plans, process routes, equipment status, overhead crane positions, and order delivery dates. Then, a scheduling state is constructed in the digital twin environment, and the hierarchical multi-agent reinforcement learning model is invoked to generate candidate scheduling actions. Finally, the scheduling scheme is distributed to the equipment control layer or the scheduling execution layer.
[0091] When equipment malfunctions, temporary orders are inserted, overhead cranes become unavailable, handover point buffers are full, or congestion occurs in a certain area, the system can rebuild the current state and call the trained policy network for rapid rescheduling. Because the decision framework of this invention processes task selection, overhead crane allocation, and conflict avoidance in a hierarchical manner, it can quickly adjust the affected decision layers when local states change, without having to completely re-search for all scheduling schemes.
[0092] In summary, this invention integrates the selection of processing resources, allocation of handling resources, and avoidance of conflicts among multiple overhead cranes in a flexible workshop with multiple tasks, multiple machines, and multiple overhead cranes into a hierarchical multi-agent reinforcement learning framework. This framework can fully express the coupling relationships between processes, machines, overhead cranes, transport paths, and overlapping areas. Compared with fixed-rule, single-agent scheduling, or ordinary multi-agent scheduling methods, this invention can reduce machine waiting time, job delays, overhead crane empty running distances, and potential collision risks, thereby improving workshop operating efficiency and safety. It has good engineering application value and promising prospects for widespread adoption.
[0093] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A flexible job shop multi-task, multi-machine, and multi-crane collaborative scheduling method based on hierarchical multi-agent deep reinforcement learning, characterized in that, Includes the following steps: S1. Obtain basic information on workpiece processes, candidate processing machines, overhead crane configuration, work areas and overlapping areas of adjacent areas in the flexible operation workshop, and establish a workpiece-machine-overhead crane collaborative scheduling model. S2. Based on the current workshop operating status, process sequence constraints, machine processing constraints, and overhead crane service area constraints, generate a candidate set of schedulable processes and machines for the current scheduling time. S3. Input the state features of the schedulable process-machine candidate set into the process-machine selection agent based on the Transformer encoder to determine the combination of the target process and the corresponding processing machine. S4. Generate material transportation tasks based on the combination of target process and corresponding processing machine. Input the candidate overhead crane status features, target workpiece features and target machine features into the overhead crane allocation agent based on cross-attention mechanism to determine the target overhead crane to perform the transportation task. S5. Based on the target crane's transportation start point, destination, estimated travel time window, and transportation route, detect the conflict relationship between the target crane and other cranes in the time and space dimensions. S6. When a spatiotemporal conflict is detected, the conflict association state is input into the crane conflict avoidance agent based on the Actor-Critic architecture, and the conflict coordination action of adjusting the yielding object, waiting time or transportation priority is output. S7. Execute joint scheduling actions that include process processing, material transportation and conflict avoidance, update the workshop operation status, update the network parameters of each agent according to the hierarchical reward and terminal reward, and repeat until all workpieces are processed. Output a collaborative scheduling scheme that includes process processing sequence, machine allocation, crane allocation, transportation path and waiting time.
2. The cooperative scheduling method according to claim 1, characterized in that, The flexible workshop includes multiple workpieces to be processed, multiple candidate processing machines, multiple overhead cranes, and multiple crane service areas; overlapping areas or virtual handover points are set between adjacent crane service areas to support continuous handling of cross-regional transportation tasks; the workpiece-machine-crane collaborative scheduling model satisfies the following constraints: Each process is handled by only one machine, and each transportation task is executed by only one overhead crane; subsequent processes must satisfy the constraints of the completion of the preceding process and the corresponding transportation task; each machine and each overhead crane can execute at most one task at any given time; when there is a conflict in the overhead crane path, the conflict is avoided by waiting, yielding, or priority adjustment.
3. The cooperative scheduling method according to claim 1, characterized in that, The collaborative scheduling model uses minimizing the maximum completion time, total delay time, crane running distance, and number of crane conflicts as joint optimization objectives to comprehensively evaluate production efficiency, delivery performance, transportation resource consumption, and transportation safety.
4. The cooperative scheduling method according to claim 1, characterized in that, The process-machine selection agent includes an embedding layer, a Transformer self-attention encoding layer, and a policy output layer. The embedding layer maps the state features of candidate process-machine combinations to a unified feature space. The Transformer encoding layer learns the correlation between different candidate combinations. The policy output layer calculates the selection probability of candidate combinations and determines the target process and processing machine combination.
5. The cooperative scheduling method according to claim 4, characterized in that, The input state features of the process-machine selection agent include the workpiece completion rate, delay risk, current machine status, machine utilization rate, machine waiting time, candidate process processing time, transportation time, target machine location, and current scheduling time.
6. The cooperative scheduling method according to claim 1, characterized in that, The overhead crane allocation agent adopts a cross-attention mechanism to calculate the task-overhead crane matching relationship based on the target transportation task feature representation and the candidate overhead crane state representation. The target transportation task feature is used as the key vector and value vector, and the candidate overhead crane state feature is used as the query vector. The candidate overhead crane matching score is obtained through attention calculation, and the target overhead crane for executing the transportation task is determined based on the matching score.
7. The cooperative scheduling method according to claim 1, characterized in that, The crane conflict avoidance agent adopts a multi-agent Actor-Critic architecture; each crane agent generates a moving or waiting action based on its own local observation state; during the training phase, the joint actions of multiple cranes are evaluated by a centralized Critic, and during the execution phase, a distributed Actor strategy is used to complete the conflict avoidance decision.
8. The cooperative scheduling method according to claim 7, characterized in that, The conflict-related status includes the degree of overlap in overhead crane mission time, the degree of overlap in transportation path space, the number of conflicts, the current position of the overhead crane, the target position, the waiting cost, and the transportation impact; the conflict avoidance actions include the current overhead crane yielding, the conflicting overhead crane yielding, delaying the transportation mission, and waiting in the overlapping area or virtual handover point.
9. The cooperative scheduling method according to claim 1, characterized in that, The tiered rewards include process-machine selection rewards, overhead crane allocation rewards, and conflict avoidance rewards. Among them, process-machine selection rewards are used to evaluate improvements in processing time, reduction in delay risk, and improvement in scheduling progress; overhead crane allocation rewards are used to evaluate transportation distance, transportation efficiency, and potential conflict changes; conflict avoidance rewards are used to evaluate conflict resolution, waiting costs, and penalties for illegal actions; and the terminal reward is calculated based on the final maximum completion time, total delay time, overhead crane running distance, number of completed processes, and number of unresolved conflicts, and then normalized.
10. The cooperative scheduling method according to claim 1, characterized in that, Each agent trains its parameters through policy gradient, advantage function estimation, experience sampling, or target network update, enabling the process-machine selection agent, crane allocation agent, and conflict avoidance agent to achieve collaborative optimization between local decision-making benefits and overall scheduling objectives.
