A real-time scheduling method for same-track double overhead traveling cranes based on joint action space deep reinforcement learning

Abstract

The application discloses a real-time scheduling method for same-track double cranes based on joint action space deep reinforcement learning, and belongs to the technical field of industrial scheduling. The method first acquires a real-time production state of double cranes; a mathematical model for real-time scheduling of double cranes is constructed, with the objective of minimizing the maximum completion time and the constraints of crane collision avoidance, workpiece process sequence and the like; then, a real-time scheduling model based on joint action space deep reinforcement learning is constructed, which adopts a joint action space mechanism; subsequently, historical data and a simulation environment are used to train the model by using a multi-agent deep reinforcement learning algorithm; finally, the real-time task and the system state are input into the trained model, so that the collaborative operation instructions of the double cranes can be output to guide the real-time operation of the production line. The application can effectively solve the path conflict, task allocation and real-time disturbance problems in the collaborative operation of double cranes, and provides reliable and efficient decision support for intelligent scheduling of the production line.

Description

Technical Field

[0001] This invention belongs to the field of industrial scheduling technology, and in particular relates to a real-time scheduling method for dual overhead cranes on the same track based on joint action space deep reinforcement learning. Background Technology

[0002] In modern intelligent manufacturing workshops, especially in heavy equipment manufacturing scenarios such as excavator chassis welding, overhead cranes, as core material handling equipment, directly determine the overall cycle time and capacity of the production line through their scheduling efficiency. Traditional scheduling methods mainly rely on manual experience or static scheduling systems, which are difficult to cope with uncertainties such as dynamic task arrival and multiple overhead crane path conflicts, easily leading to idle resources, task delays, and even safety hazards.

[0003] To improve the intelligence level of scheduling, existing technologies mainly explore the following directions: First, using reinforcement learning frameworks to model overhead crane scheduling and using historical data for training to achieve real-time decision-making based on environmental conditions; Second, focusing on the path planning problem of a single overhead crane and using algorithms such as Q-learning to find the shortest path to avoid obstacles in complex environments; Third, for multi-agent deep reinforcement learning scenarios, using multi-agent deep reinforcement learning for centralized training and distributed execution to handle cooperation and conflicts between overhead cranes, such as implementing anti-blocking path planning for overhead cranes in semiconductor automated material handling systems (AMHS), or building a multi-agent DRL model with overhead cranes as agents in metallurgical workshops to optimize global scheduling.

[0004] However, the above methods still have significant limitations: First, they fail to effectively integrate two extremes: direct workstation actions (which are precise but have a large search space and lack domain knowledge guidance) and scheduling rule actions (which suffer from broken causal chains and poor learning stability). Second, most methods rely on fixed environment models or large amounts of offline historical data, resulting in insufficient adaptability and real-time decision-making when facing real-time dynamic changes in the production line. Finally, for dual overhead crane systems, existing research does not comprehensively consider complex constraints such as path conflicts, safety distances, and task coordination, and lacks an efficient decision-making mechanism that can balance direct workstation actions and rule-based scheduling actions, thus limiting their practicality and generalization performance in complex industrial scenarios.

[0005] To address the aforementioned problems in existing technologies, there is an urgent need to propose a real-time scheduling method based on deep reinforcement learning of the production line joint motion space. Summary of the Invention

[0006] To address the aforementioned technical problems, this invention provides a real-time scheduling method for dual overhead cranes on the same track using joint action space deep reinforcement learning, comprising the following steps: The real-time production status of the production line is obtained, including information on processing equipment, work-in-process status, and the operating status of the two overhead cranes. Based on the real-time production status, a real-time scheduling mathematical model for dual overhead cranes is constructed with the objective function of minimizing the maximum completion time, and includes constraints such as the safe operating distance between overhead cranes, the process sequence of workpieces, and dynamic task allocation. Based on the aforementioned real-time scheduling mathematical model for dual overhead cranes, a deep reinforcement learning real-time scheduling model based on a joint action space is constructed. The deep reinforcement learning real-time scheduling model adopts a joint action space mechanism of direct workstation action subspace and scheduling rule action subspace to perform real-time scheduling of the collaborative operation of dual overhead cranes. Using historical production data and a simulation environment, a multi-agent deep reinforcement learning algorithm is employed to train the deep reinforcement learning real-time scheduling model. The real-time collected production tasks and system status are input into the trained deep reinforcement learning real-time scheduling model, which outputs dual overhead crane collaborative operation instructions to guide the real-time operation of the production line.

[0007] Optionally, the real-time production status of the production line can be obtained, including: The processing equipment information includes equipment status and remaining processing time; The work-in-process status information includes workpiece identifier, current workstation, set of completed processes, sequence of processes to be executed, and delivery deadline. The operating status information of the two overhead cranes includes the crane number, current position coordinates, load status, and distance between the cranes.

[0008] Optionally, the constraints including the safe operating distance between workstations, the workpiece process sequence, and dynamic task allocation specifically include: At any given time and at any given workstation, only one overhead crane can perform operations at that workstation. At any given time, the distance between two overhead cranes must be greater than the preset minimum safe distance; Each crane must be positioned within the length of the track; The start time of any workpiece at the workstation shall not be later than the completion time. At any workstation, the task completion time equals the start time plus the processing time; The processes to be performed on any workpiece follow a predefined process sequence.

[0009] Optionally, in constructing a deep reinforcement learning real-time scheduling model based on a joint action space, the dual-crane cooperative scheduling problem is described as a partially observable Markov decision process, specifically including: Define the state transition process of the system and set up a candidate action update mechanism. When the system simulation time progresses to the point where both overhead cranes are in an idle state, if there is a task to be transported that is in a waiting state, the task to be transported is converted into a transport action of the overhead crane and added to the candidate action space as one of the candidate actions in the state transition process.

[0010] Optionally, in constructing a deep reinforcement learning real-time scheduling model based on a joint action space, the joint action space includes: The direct workstation action subspace consists of a set of discrete workstation identifier actions, each action corresponding to the control command for the overhead crane to move directly to the specified workstation; The scheduling rule action subspace consists of a set of discrete scheduling rule identifiers. Each action corresponds to selecting a scheduling rule from a preset scheduling rule set, which can calculate the target workstation based on the real-time status. Optionally, the scheduling rules included in the scheduling rule action subspace include at least: NN rules, FIFO rules, SPT rules, EDD rules, and CR rules.

[0011] Optionally, the construction of a deep reinforcement learning real-time scheduling model based on a joint action space also includes an action parsing mechanism: When the action output by the intelligent agent belongs to the scheduling rule action subspace, the system calls the corresponding scheduling rule function according to the action identifier, calculates the specific target workstation based on the current real-time production status, and then moves the overhead crane to the target workstation to perform loading and unloading operations.

[0012] Optionally, a reward calculation mechanism is also included in the construction of a deep reinforcement learning real-time scheduling model based on a joint action space: When an agent selects a direct workstation action and that workstation is not the current workstation, a success reward is given. When an agent selects a direct workstation action and that workstation is the current workstation, a negative reward is given. When an agent selects a scheduling rule action, it is given a higher success reward than a direct workstation action. When the intelligent agent controls the crane to move, a movement penalty related to the movement distance is applied to improve the crane's handling efficiency.

[0013] Optionally, in constructing a deep reinforcement learning real-time scheduling model based on a joint action space, the deep reinforcement learning scheduling model adopts a multi-objective composite reward function, and the total reward is represented as the weighted sum of the completion reward components and the action reward components of all scheduled tasks. The reward component for completing all scheduled tasks aims to minimize the maximum completion time. The action reward component is the action reward for the agent to select direct workstation actions and scheduling rule actions. The cumulative reward discounts future rewards to the current time step through a discount factor.

[0014] On the other hand, the present invention also provides an electronic device including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0015] Compared with the prior art, the present invention has the following advantages and technical effects: This invention collects real-time data on the operating status of processing equipment, work-in-process, and dual overhead cranes. Based on this data, it constructs a mathematical model that aims to minimize the maximum completion time and incorporates multiple constraints such as safety distance and process sequence. This approach transforms the dynamic and uncertain production environment into a computable and optimizable decision problem, providing a stable and physically realistic problem framework for subsequent intelligent algorithms and laying the foundation for real-time scheduling.

[0016] This invention designs a joint action space comprising direct workstation actions and scheduling rule actions, and performs unified learning through flattened coding. This design enables the agent to perform precise terminal control and also invoke domain knowledge (such as prioritizing tasks with the shortest processing time), with the system parsing rules and calculating specific workstations in real time. This mechanism effectively integrates the flexibility of data-driven approaches with the reliability of rule-based experience, guiding the agent to quickly learn efficient and easily understandable scheduling strategies.

[0017] This invention employs a multi-agent collaborative training framework, allowing agents representing each overhead crane to make independent decisions and learn under a shared, centralized evaluation. A composite reward function is designed that integrates rewards for completing all scheduling tasks and actions. This approach enables agents not only to optimize their individual task execution efficiency during the learning process but also to spontaneously learn collision avoidance, maintaining safe distances, and collaborative handling. Ultimately, it outputs dual-crane collaborative scheduling instructions that fully meet complex process and safety constraints.

[0018] This invention constructs a complete closed-loop system from real-time data input to online model decision-making and finally to instruction-driven execution. By deploying a trained deep reinforcement learning model on the online system, it can continuously receive real-time production status and instantly output instructions for collaborative operation of two overhead cranes. This closed-loop automated process replaces traditional manual scheduling or static scheduling, enabling the production line to autonomously respond to dynamic disturbances, significantly improving overall operational efficiency, resource utilization, and system response speed, and achieving a fundamental intelligent upgrade in real-time scheduling management. Attached Figure Description

[0019] The accompanying drawings, which form part of this application, are used to provide a further understanding of this application. The illustrative embodiments and descriptions of this application are used to explain this application and do not constitute an undue limitation of this application. In the drawings: Figure 1 This is an overall flowchart of the real-time scheduling method based on Joint Action Space Deep Reinforcement Learning (JAS-DRL) according to an embodiment of the present invention. Figure 2 This is a schematic diagram of the model training and optimization process in an embodiment of the present invention; Figure 3 This is a diagram of the collaborative operation trajectory of two overhead cranes output by the method described in this embodiment of the invention. Figure 4 This is a comparison diagram of the running trajectory of a single overhead crane under various scheduling strategies according to embodiments of the present invention; Figure 5 This is a flowchart illustrating the specific implementation of the real-time scheduling method for excavator travel frame welding production line according to an embodiment of the present invention. Detailed Implementation

[0020] It should be noted that, unless otherwise specified, the embodiments and features described in this application can be combined with each other. This application will now be described in detail with reference to the accompanying drawings and embodiments.

[0021] It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0022] Example 1 like Figure 5 As shown, this embodiment provides a real-time scheduling method for dual overhead cranes on the same track using joint action space deep reinforcement learning, including the following steps: S1: Obtain the real-time production status of the production line, which includes processing equipment information, work-in-process status information, and dual overhead crane operation status information; S2: Based on the real-time production status, construct a real-time scheduling mathematical model for dual overhead cranes with the objective function of minimizing the maximum completion time, and including constraints such as the safe operating distance between overhead cranes, the process sequence of workpieces, and dynamic task allocation. S3: Based on the real-time scheduling mathematical model of the two overhead cranes, a real-time scheduling model of the two overhead cranes based on the joint action space is constructed. The collaborative scheduling problem of the two overhead cranes is described as a partially observable Markov decision process. A joint action space mechanism that integrates direct workstation actions and scheduling rule actions is designed to perform real-time scheduling of the collaborative operation of the two overhead cranes. S4: Using historical production data and simulation environment, a multi-agent deep reinforcement learning algorithm is used to collaboratively train and optimize the deep reinforcement learning real-time scheduling model. The processing equipment status, workpiece status and crane operation status are extracted in multiple dimensions and used as model input. S5: Input the real-time collected production tasks and system status into the trained deep reinforcement learning real-time scheduling model, and output dual overhead crane collaborative operation instructions to guide the real-time operation of the production line.

[0023] The feasible mathematical model for real-time scheduling of dual overhead cranes in step S1 is based on the information of processing equipment, the status of work-in-process, and the operating status of the dual overhead cranes. The information of processing equipment includes equipment status and remaining processing time. The status of work-in-process includes workpiece ID, current workstation, set of completed processes, sequence of processes to be executed, and delivery deadline. The operating status of the dual overhead cranes includes crane number, current position coordinates, load status, and distance between the cranes.

[0024] The feasible constraints in step S2 include: At any given time and at any given workstation, only one overhead crane can perform operations at that workstation. At any given time, the positional difference between the two overhead cranes must be greater than the minimum safe distance; Each overhead crane must be located within the track area; The start time of any part at the workstation must not be later than the completion time. At any workstation, the task completion time is equal to its start time plus the processing time. The processes to be performed on any workpiece must strictly follow the predefined process sequence.

[0025] In feasible step S3, the dual-crane coordinated scheduling problem is described as a partially observable Markov decision process. The candidate action update mechanism in the state transition process is as follows: when the system simulation time advances to the point where both cranes are in an idle state, if there is a task to be transported that is in a waiting state, the task to be transported is converted into a crane transport action and added to the candidate action space, becoming one of the candidate actions in the state transition process.

[0026] Implementable, the deep reinforcement learning real-time scheduling model constructed in step S3 adopts a joint action space, which includes: a direct workstation action subspace, consisting of a set of discrete workstation identifier actions, each action corresponding to the control command for the overhead crane to move directly to the specified workstation; and a scheduling rule action subspace, consisting of a set of discrete scheduling rule identifier actions, each action corresponding to selecting a rule from a preset scheduling rule set, which calculates the target workstation based on the real-time status.

[0027] Furthermore, the dual overhead crane deep reinforcement learning scheduling model based on the joint action space has an action space designed as a composite structure including a direct workstation action subspace and a scheduling rule action subspace; the direct workstation action subspace refers to the agent directly outputting the specific workstation number to which the overhead crane will go; the scheduling rule action subspace refers to the agent selecting a preset scheduling rule, and the system calculating the target workstation according to the rule.

[0028] Furthermore, the scheduling rule action subspace includes at least one of the following scheduling rules: NN rule, FIFO rule, SPT rule, EDD rule, and CR rule.

[0029] The joint action space maps direct workstation actions and scheduling rule actions to a single discrete action space through flattened encoding, and performs policy learning through a unified deep reinforcement learning network.

[0030] Implementable step S3 also includes an action parsing mechanism: when the action output by the agent belongs to the scheduling rule action subspace, the system calls the corresponding scheduling rule function according to the action identifier, calculates the specific target workstation based on the current real-time production status, and then moves the overhead crane to the target workstation to perform loading and unloading operations.

[0031] Implementable step S3 also includes a reward calculation mechanism: when the agent selects a direct workstation action and that workstation is not the current workstation, a success reward is given; when the agent selects a direct workstation action and that workstation is the current workstation, a negative reward is given; when the agent selects a scheduling rule action, a success reward higher than that for a direct workstation action is given; when the agent controls the crane to move, a movement penalty related to the movement distance is given to improve the crane's handling efficiency.

[0032] In this embodiment, the deep reinforcement learning real-time scheduling model adopts a multi-objective composite reward function. The total reward is represented as the weighted sum of the completion reward component and the action reward component of all scheduled tasks. The completion reward component of all scheduled tasks aims to minimize the maximum completion time. The action reward component is the action reward for the agent to select direct workstation actions and scheduling rule actions. The cumulative reward discounts future rewards to the current time step through a discount factor.

[0033] Compared with the prior art, this embodiment has the following advantages: This embodiment constructs a joint action space that integrates direct control actions and scheduling rule actions, and introduces additional rewards for successful rule execution. This effectively guides the agent to actively invoke engineering-verified scheduling rules (such as SPT, EDD, etc.) under complex or high-load conditions. It retains the adaptive exploration capability of deep reinforcement learning, and significantly improves the interpretability, stability and generalization performance of the policy. It overcomes the dual defects of black-box decision-making in traditional pure data-driven methods and instability in pure rule-based methods.

[0034] The real-time scheduling mathematical model constructed in this embodiment not only takes minimizing the maximum completion time as its core optimization objective, but also comprehensively incorporates multiple complex constraints such as the safe operating distance between overhead cranes, the strict process sequence of workpieces, and dynamic task allocation. In particular, by describing the dual-crane collaborative scheduling problem as a partially observable Markov decision process, the model can more accurately characterize the dynamic interaction and path conflicts of the dual-crane system, thereby generating an efficient and conflict-free collaborative scheduling scheme that satisfies all process and safety constraints.

[0035] This embodiment constructs a complete closed loop from real-time state awareness (S1) to model decision-making (S5). By acquiring multi-dimensional state information of the processing equipment, work-in-process, and dual overhead cranes in real time, and extracting features from this information, the system inputs it into a trained deep reinforcement learning model. The model can then instantly output collaborative operation instructions for the dual overhead cranes. This real-time state-based decision-making mechanism enables the system to quickly adapt to dynamic disturbances such as the insertion of new tasks, changes in equipment state, and updates to crane positions. It achieves a leap from static offline scheduling to dynamic online real-time scheduling, significantly improving the production line's ability to cope with uncertainties.

[0036] Example 2 In this embodiment, the real-time scheduling problem of the excavator undercarriage welding line is described as follows: There are n undercarriage welding tasks, each task is further refined through a process route, and a task is decomposed into multiple operations to be executed in a strict sequence. The undercarriage welding workshop is equipped with m functional workstations (such as positioning, welding, and inspection), and 2 overhead cranes operating on a common rail. Each operation must be completed at its corresponding workstation, and material transfer between operations must be completed by the overhead cranes. The real-time scheduling process involves allocating overhead cranes to each transfer request and planning conflict-free actions, ultimately forming a complete real-time scheduling scheme.

[0037] This embodiment provides a real-time scheduling method for the welding line of an excavator traveling frame with a dual overhead crane configuration on the same track, including the following steps: S1: Obtain the real-time production status of the excavator traveling frame welding line, the real-time production status including processing equipment information, work-in-process status information and dual overhead crane operation status information; S2: Construct a real-time scheduling mathematical model for the welding line of the double overhead crane excavator traveling frame. The mathematical model includes constraints for handling overhead crane path conflicts, emergency orders and dynamic task allocation, as well as an objective function aimed at minimizing the maximum completion time.

[0038] Furthermore, the basic definition of the workshop real-time scheduling mathematical model is constructed around the information of processing equipment, the status information of work-in-process, and the operating status information of the two overhead cranes. The information of processing equipment includes workstation number, workstation type, current occupancy status, and remaining processing time. The status information of work-in-process includes workstation ID, current workstation, set of completed processes, sequence of processes to be executed, process constraints, and delivery deadline. The operating status information of the two overhead cranes includes crane number, current position coordinates, load status, and minimum safe distance between the cranes.

[0039] Furthermore, the constraints include: At any given time and at any given workstation, only one overhead crane can perform operations at that workstation. At any given time, the positional difference between the two overhead cranes must not be less than the minimum safe distance; Each overhead crane must be located within the track area; The start time of any part at the workstation must not be later than the completion time. At any workstation, the task completion time is equal to its start time plus the processing time. The processes to be performed on any workpiece must strictly follow the predefined process sequence.

[0040] Specifically, the constraints are expressed as follows: ; ; ; ; ; ; in This indicates whether crane i is performing an operation at workstation j at time t; This indicates the position of crane i at time t. This represents the minimum safe distance between two overhead cranes; L is the total length of the crane track. and These represent the start and finish times of part k at the workstation, respectively. and These represent the start and finish times of part k at station j, respectively. Let be the processing time of part k at station j.

[0041] S3: Construct a real-time scheduling model for dual overhead cranes based on a joint action space, describing the dual overhead crane collaborative scheduling problem as a partially observable Markov decision process, and designing a joint action space mechanism that integrates the direct workstation action subspace and the scheduling rule action subspace to perform real-time scheduling of the collaborative operation of dual overhead cranes. Furthermore, the action space of the model includes multiple preset scheduling rules and direct workstation control instructions. The improved multi-agent deep Q-network selects actions based on the current state and generates specific overhead crane operation instructions based on the selected actions.

[0042] Furthermore, the preset scheduling rules include: nearest neighbor rule (NN), first-come, first-served rule (FIFO), shortest processing time rule (SPT), earliest delivery date rule (EDD), and urgency rate rule (CR); the direct workstation control instructions include actions that specify target workstation numbers 0 to 5.

[0043] (1) State space: In the constructed intelligent real-time scheduling framework, the system state space is formalized as a multi-dimensional real-valued vector to comprehensively characterize the dynamic environment of crane operation and workstation work. This state vector is defined as: ; in Indicates the physical location (in meters) of the i-th overhead crane. Indicates its cargo status (0 for empty, 1 for loaded). This represents the demand status of the j-th workstation (-1 indicates that goods need to be picked up, 0 indicates that the workstation is idle, and 1 indicates that goods need to be placed), while and These represent the average waiting time (minutes) and average utilization rate for all workstations, respectively. To improve the training stability and convergence of the deep reinforcement learning model, all state components are normalized: the position variable is linearly scaled to the total track length L meters. The average waiting time was dynamically normalized using the 95th percentile of historical samples (up to 180), while the utilization rate was directly truncated to... The design balances physical interpretability with the numerical stability of neural network inputs, providing an agent with a sufficient and structured foundation for environmental perception.

[0044] (2) Action space: The action space unifies direct workstation selection and rule-driven decision-making into a single action space. Through a single decision, the agent can specify a concrete workstation or select an abstract rule. After rule selection, the rule function performs a secondary calculation based on the current state to determine the final target workstation.

[0045] The motion space design is as follows: ; ; ; Action space It consists of two parts, namely direct action and Scheduling rules and actions; Indicates the number of workstations; This indicates the number of actions in the scheduling rule.

[0046] This embodiment also proposes a scheduling mechanism based on action parsing. The action selected by the agent is processed by a parsing function. Mapping to a specific target workstation: If it's a direct workstation action, the target workstation is the action index; if it's a scheduling rule action, the corresponding rule function is called. The optimal workstation is calculated based on the current state. The action parsing function and scheduling rule function are as follows: ; ; ; ; ; ; in This represents the NN rule, which selects the workstation closest to the current position of the overhead crane; This represents the SPT rule, which selects the task with the shortest processing time. This indicates the EDD rule, which prioritizes the task with the earliest deadline. This refers to the CR rule, which sorts processes by urgency. It is usually defined as the ratio of remaining time to remaining processing time. This indicates a FIFO rule, which selects the task with the earliest arrival time for processing.

[0047] (3) Reward function: This embodiment designs an adaptive reward mechanism to guide the agent in effectively balancing direct workstation actions and scheduled rule actions. The reward function is defined as: ; in Basic rewards; This is an additional reward coefficient; For indicator functions; Indicates whether the action was successfully executed.

[0048] The additional reward mechanism aims to encourage agents to try using scheduling rules, while avoiding their inhibitory effect on direct actions through a small additional reward (about 10% of the base reward), thereby achieving an autonomous balance between the two types of actions.

[0049] S4: Using historical production data and simulation environment, a multi-agent deep reinforcement learning algorithm is used to collaboratively train and optimize the deep reinforcement learning real-time scheduling model. Furthermore, the processing equipment status, workpiece status, and crane operating status are extracted using multi-dimensional features and used as model inputs. MADQN (Multi-Agent Deep Q-Network) is adopted as the basic framework, configuring an independent policy network (Actor) for each crane, but sharing a centralized critic network to achieve "independent decision-making and collaborative evaluation." The policy network consists of a three-layer fully connected neural network with an output dimension of 12.

[0050] Furthermore, a Prioritized Experience Replay mechanism is employed during training, assigning higher sampling weights to samples with high TD-error (such as conflict events) to accelerate the learning of key experiences. The optimizer used is Adam, with an initial learning rate of 0.001 and a discount factor of [missing information]. =0.99.

[0051] S5: Input the real-time collected production tasks and system status into the trained deep reinforcement learning real-time scheduling model, and output dual overhead crane collaborative operation instructions to guide the real-time operation of the production line. The JAS-DRL (Joint ActionSpace-Deep Reinforcement Learning) agent training process is as follows: Figure 1 As shown.

[0052] Taking a single instance as an example, this instance contains 12 order tasks, which arrive randomly, 6 functional workstations, and 2 common-rail overhead cranes. At any given time, a workstation can process at most one operation. A specific operation can only be executed after all its preceding operations are completed. At any given time, the distance between two overhead cranes cannot be less than a safety threshold. The newly added tasks, workstations, and crane statuses are read into the algorithm in database form, providing the scheduling result. The instances used in the training and testing of this embodiment are extracted from historical task data from the actual workshop and are divided into training and testing sets. The algorithm's performance on the testing set after 20,000 training iterations is as follows: Figure 2As shown in the figure. After 20,000 training iterations, this embodiment observed that the reward gradually stabilized and essentially converged. At this point, this embodiment considers the model suitable for testing and application.

[0053] This embodiment presents the results of the real-time scheduling scheme as a dual-crane operation trajectory diagram, as shown below. Figure 3 As shown, the results clearly demonstrate the operational tasks, target workstations, and movement trajectories of each overhead crane within each time period, effectively verifying the superiority of this embodiment in handling collision avoidance and dynamic disturbances between two overhead cranes. Furthermore, this embodiment also compares the trajectory performance of the second overhead crane using four scheduling strategies (JAS-DRL, NN, EDD, CR) over a 60-minute period. The results are as follows... Figure 4 As shown: JAS-DRL's scheduling strategy enables flexible scheduling of the overhead crane among multiple workstations, making full use of resources; the scheduling rule strategies (EDD, CR) have similar and relatively fixed trajectories, which are prone to getting trapped in local optima; the NN strategy moves between the buffer zone, the second base frame finishing robot and the traveling frame assembly workstation, using only distance as the scheduling basis.

[0054] The results of this example verify that the method proposed in this embodiment can effectively handle the problems of path planning, conflict avoidance and dynamic task allocation in the collaborative operation of two overhead cranes, and finally generate efficient and conflict-free real-time scheduling instructions, which significantly improves the material handling efficiency and overall production cycle of the heavy equipment manufacturing production line.

[0055] Example 3 This embodiment provides an electronic device, including a memory, a processor, and a computing program stored in the memory and executable on the processor, wherein the processor implements the method when executing the computing program.

[0056] The above are merely preferred embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A real-time scheduling method for dual overhead cranes on the same track using joint action space deep reinforcement learning, characterized in that, Includes the following steps: The real-time production status of the production line is obtained, including information on processing equipment, work-in-process status, and the operating status of the two overhead cranes. Based on the real-time production status, a real-time scheduling mathematical model for dual overhead cranes is constructed with the objective function of minimizing the maximum completion time, and includes constraints such as the safe operating distance between overhead cranes, the process sequence of workpieces, and dynamic task allocation. Based on the aforementioned real-time scheduling mathematical model for dual overhead cranes, a deep reinforcement learning real-time scheduling model based on a joint action space is constructed. The deep reinforcement learning real-time scheduling model adopts a joint action space mechanism composed of direct workstation actions and scheduling rule actions to perform real-time scheduling of the collaborative operation of dual overhead cranes. Using historical production data and a simulation environment, a multi-agent deep reinforcement learning algorithm is employed to train the deep reinforcement learning real-time scheduling model. The real-time collected production tasks and system status are input into the trained deep reinforcement learning real-time scheduling model, which outputs dual overhead crane collaborative operation instructions to guide the real-time operation of the production line.

2. The method according to claim 1, characterized in that, Obtain the real-time production status of the production line, including: The processing equipment information includes equipment status and remaining processing time; The work-in-process status information includes workpiece identifier, current workstation, set of completed processes, sequence of processes to be executed, and delivery deadline. The operating status information of the two overhead cranes includes the crane number, current position coordinates, load status, and distance between the cranes.

3. The method according to claim 1, characterized in that, The constraints, including safe operating distance between workstations, workpiece process sequence, and dynamic task allocation, specifically include: At any given time and at any given workstation, there can be at most one overhead crane performing operations at that workstation. At any given time, the distance between two overhead cranes must be greater than the preset minimum safe distance; Each crane must be positioned within the length of the track; The start time of any workpiece at the workstation shall not be later than the completion time. At any workstation, the task completion time is equal to the sum of the start time and the processing time; The processes to be performed on any workpiece follow a predefined process sequence.

4. The method according to claim 1, characterized in that, In constructing a deep reinforcement learning real-time scheduling model based on a joint action space, the dual-crane cooperative scheduling problem is described as a partially observable Markov decision process, specifically including: Define the state transition process of the system and set up a candidate action update mechanism. When the system simulation time progresses to the point where both overhead cranes are in an idle state, if there is a task to be transported that is in a waiting state, the task to be transported is converted into a transport action of the overhead crane and added to the candidate action space as one of the candidate actions in the state transition process.

5. The method according to claim 1, characterized in that, In constructing a deep reinforcement learning real-time scheduling model based on a joint action space, the joint action space includes: The direct workstation action subspace consists of a set of discrete workstation identifier actions, each action corresponding to the control command for the overhead crane to move directly to the specified workstation; The scheduling rule action subspace consists of a set of discrete scheduling rule identifiers. Each action corresponds to selecting a scheduling rule from the preset scheduling rule set, and the target workstation in the current real-time state is calculated based on the rule.

6. The method according to claim 5, characterized in that, The scheduling rules included in the scheduling rule action subspace include at least the following: nearest neighbor rule, first-come-first-served rule, shortest processing time rule, earliest delivery date rule, and urgency rate rule.

7. The method according to claim 5, characterized in that, The construction of a deep reinforcement learning scheduling model based on a joint action space also includes an action parsing mechanism: When the action output by the intelligent agent belongs to the scheduling rule action subspace, the system calls the corresponding scheduling rule function according to the action identifier, calculates the specific target workstation based on the current real-time production status, and moves the overhead crane to the target workstation to perform loading and unloading operations.

8. The method according to claim 7, characterized in that, The construction of a deep reinforcement learning scheduling model based on a joint action space also includes a reward calculation mechanism: When an agent selects a direct workstation action and that workstation is not the current workstation, a success reward is given. When an agent selects a direct workstation action and that workstation is the current workstation, a negative reward is given. When an agent selects a scheduling rule action, it is given a higher success reward than a direct workstation action. When the agent controls the crane to move, a movement penalty related to the distance moved is imposed.

9. The method according to claim 1, characterized in that, In constructing a deep reinforcement learning scheduling model based on a joint action space, the deep reinforcement learning scheduling model adopts a multi-objective composite reward function, and the total reward is represented as the weighted sum of the completion reward components and the action reward components of all scheduled tasks. The reward component for completing all scheduled tasks aims to minimize the maximum completion time. The action reward component is the action reward for the agent to select direct workstation actions and scheduling rule actions. The cumulative reward discounts future rewards to the current time step through a discount factor.

10. An electronic device comprising a memory, a processor, and a computing program stored in the memory and executable on the processor, characterized in that, When the processor executes the computing program, it implements the method of any one of claims 1-9.

Images