A shipyard steel plate warehouse-out scheduling method based on hierarchical reinforcement learning

Abstract

The present application relates to a kind of shipyard steel plate warehouse scheduling method based on layered reinforcement learning, belong to shipyard steel plate yard intelligent scheduling optimization technical field.For the problem that existing scheduling algorithm does not consider long-term jam risk, low efficiency in large-scale scene solving, the present application first constructs the mixed integer linear programming model fusing the number of day's unstacking and long-term blocking degree;Further design layered near strategy optimization framework, wherein high-level agent is responsible for decision target steel plate export sequence, low-level agent is responsible for assigning obstacle steel plate unstacking position, realize double-layer decoupling decision by nested interaction.The present application introduces action mask mechanism in high-low layer network to shield invalid action, and adapts different scale yards by feature zero processing.Application shows that the method can output scheduling scheme within seconds, while ensuring the lowest daily unstacking cost, effectively reducing the jam risk of subsequent operation, improving the whole cycle logistics efficiency of yard.

Description

Technical Field

[0001] This invention belongs to the field of intelligent scheduling and optimization technology for shipyard steel plate storage yards, specifically involving a shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning. Background Technology

[0002] With the development of the shipbuilding industry and industrial intelligent optimization technology, the shipbuilding field has placed higher demands on efficient intelligent scheduling methods for steel plate materials. By intelligently planning the target steel plate outbound sequence and the location of barrier plate repositioning, it is possible not only to significantly reduce ineffective repositioning operations on-site and accelerate the material cutting process, but also to predict potential long-term blockages in the stacking locations, achieving long-term effective management of the stockpile status. Considering the large scale of steel plate stockpiles in shipyards and the high number of outbound shipments per day, designing an intelligent scheduling strategy for steel plate outbound shipments that is suitable for large-scale stockpiles while taking into account both daily repositioning costs and long-term blockage risks is of great application value.

[0003] The existing technology is as follows:

[0004] 1. Chinese patent application number CN202111303880.6, entitled "A Material Discharge Method for Steel Plate Stockyards in Shipyards," uses the steel plate stockpile quantity as the source node and the material discharge plan as the target node, employing an ant colony algorithm for path planning. However, this method fails to incorporate the last-in-first-out (LIFO) stacking constraint of the steel plate stockyard into its stacking optimization logic, making it impossible to quantify the long-term congestion costs caused by daily stacking operations and the steel plates awaiting retrieval the following day. Furthermore, traditional ant colony algorithms rely on random mechanisms for optimization, which can easily get trapped in local optima in large-scale stockpile scenarios, resulting in time-consuming scheduling and solutions. This makes it impossible to simultaneously meet the requirements of real-time on-site scheduling and the optimization needs of stockyard layout under continuous production conditions.

[0005] 2. Chinese patent application number CN202511742671.X, entitled "A Dynamic Optimization Method for Shipyard Steel Plate Stacking Yards Based on Spatiotemporal Coupled Risk Propagation," proposes a dynamic "flipping risk field" model, combining adaptive greedy algorithms and path reconnection local search to complete the steel plate warehousing stacking location planning. However, this method only optimizes the stacking layout for the steel plate warehousing stage, failing to address the stacking optimization decision-making for the steel plate warehousing scenario, and thus cannot simultaneously optimize the daily target steel plate warehousing sequence and obstacle plate stacking location allocation. Furthermore, this method employs a traditional iterative search algorithm, which results in slow online solution speeds in large-scale stacking yards, making it difficult to achieve second-level real-time scheduling output.

[0006] It is evident that in shipyard steel plate stockpile material scheduling applications, balancing daily repacking costs with long-term stockpile congestion risks can significantly improve the continuous production efficiency of shipbuilding. However, with the increase in stockpile positions and the number of steel plates, the solution space during scheduling exhibits an explosive growth trend. This is especially true in large shipyard raw material stockpiles with densely distributed stockpiles and a large daily outbound steel plate volume. Scheduling algorithms need to simultaneously optimize the steel plate outbound timing and allocate obstructed steel plate repacking positions to output feasible steel plate outbound solutions. This undoubtedly increases the solution time of traditional algorithms significantly, making it difficult to meet the real-time scheduling requirements on-site. Secondly, since steel plate stockpiles strictly adhere to the last-in-first-out (LIFO) stacking constraint, existing scheduling optimization methods are mostly single-cycle short-sighted optimizations, posing a significant challenge to the long-term stockpile layout optimization effect. Summary of the Invention

[0007] The purpose of this invention is to overcome the shortcomings of existing technologies and provide a shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning. By using a hierarchical decoupling approach, the scheduling problem is divided into a two-layer decision architecture. It comprehensively considers the outbound and stacking costs on the same day and the potential for congestion of steel plates to be picked up the next day, and establishes a dual-objective collaborative optimization mechanism. Under the premise of significantly reducing the overall scheduling cost of the yard, it greatly shortens the scheduling solution time, thereby improving the efficiency of the entire life cycle logistics operation of the shipyard steel plate yard.

[0008] To achieve the above objectives, the technical solution of the present invention is: a shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning, comprising:

[0009] Collect basic operating data of the shipyard's steel plate stockpile and construct the original dataset for stockpile scheduling.

[0010] Based on basic working condition data, a mathematical model for the dispatching of steel plates out of the shipyard is established, including an objective function and constraints. The objective function includes a term that minimizes the total number of stacking operations on the day and a term that minimizes the degree of obstruction of the stacking position when the dispatching is terminated.

[0011] A hierarchical near-end strategy optimization scheduling framework is constructed. The framework includes a high-level agent and a low-level agent. The high-level agent is used to output the target steel plate outbound sequence, and the low-level agent is used to allocate the stacking position of the obstacle steel plates according to the target steel plate outbound sequence.

[0012] Design the state feature vectors, action spaces, and reward functions for high-level and low-level intelligent agents;

[0013] The high-level agent and the low-level agent are jointly trained offline based on the original data set of the yard scheduling to obtain the trained scheduling model.

[0014] Input real-time operating data into the trained scheduling model to output a steel plate outbound scheduling scheme.

[0015] Furthermore, the basic operating data includes: the capacity of the storage yard, the stacking order of steel plates in each storage yard, the list of steel plates to be shipped out on the current day, the distribution of steel plates to be picked up the next day, and the upper limit of the stacking height.

[0016] Furthermore, the decision variables in the shipyard steel plate outbound scheduling mathematical model include: the time occupancy state variable of the steel plate in the stack slot. Variables of steel plate removal action Variables for steel plate movement 1. Steel plate outbound completion status variable and blocking indicator variables ;in, Let i, k, and c be the time steps, representing the i-th steel plate, the k-th stack position, and the c-th layer slot position, respectively.

[0017] The constraints include constraints on the spatial layout and state evolution of the storage yard, operational logic constraints, and obstruction level constraints.

[0018] Furthermore, the hierarchical near-end strategy optimization scheduling framework adopts nested interaction logic, where a single decision by a high-level agent corresponds to a round of stack-turning sub-tasks by a low-level agent, until the target steel plate is exposed to the top of the stack and the outbound process is completed.

[0019] Furthermore, the state feature vector of the high-level intelligent agent is composed of the three-dimensional features of all stack positions, and the three-dimensional features include: the proportion of unplanned steel plate blockage. The proportion of steel plates blocked before being retrieved and stack release degree ;

[0020] The action space of the high-level intelligent agent is to select the target stack number to be shipped out. ;

[0021] The reward function of the high-level intelligent agent Defined as:

[0022]

[0023] in, To determine the total number of obstacle steel plates that need to be re-stashed in step T, This is the immediate reward for the lower-level intelligent agent at time t.

[0024] Furthermore, the stack release degree The calculation method is as follows:

[0025]

[0026] Where x is the number of steel plates remaining in the stack after the top steel plate is removed; the coefficient a is determined according to the following rules: 1) when the remaining plates include the target steel plates to be shipped out on the same day, a=0; 2) when the remaining plates only contain steel plates to be shipped out later and no target steel plates to be shipped out on the same day, a=0.5; 3) when the remaining plates contain neither target steel plates to be shipped out on the same day nor steel plates to be shipped out later, a=1.

[0027] Furthermore, the state feature vector of the low-level intelligent agent includes the identifier of the stacked steel plate. Stack space utilization rate Blockage intensity The degree of impact of single-step stacking Extended stacking impact ;

[0028] The action space of the lower-level intelligent agent is to select the candidate stacking position number for storing the current obstacle steel plate to be overturned. .

[0029] Furthermore, the degree of impact of the single-step stacking... The calculation method is as follows:

[0030]

[0031] in Represents the highest outbound priority number; , Steel plates to be stacked The priority number and the minimum priority number of the steel plate in stack position k, where k and j are the candidate stack position and the target stack position, respectively.

[0032] Furthermore, the reward function of the low-level agent is configured as follows: if, after the stacking, there is no steel plate with an earlier outbound time below the stacked steel plate, a positive reward is given. If the stacked steel plates are stored on top of steel plates that were issued earlier, a negative reward will be given. .

[0033] Furthermore, an action masking mechanism is introduced in the policy network output layer of both high-level and low-level intelligent agents to shield invalid stacking position selection actions that do not conform to the on-site operation rules.

[0034] When the actual number of stack positions in the calculation is less than the preset maximum number of stack positions At that time, zero-padding is performed on the insufficient parts of the state feature vectors of high-level and low-level intelligent agents to fix the input dimension of the neural network.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] (1) For the shipyard steel plate storage yard scenario with dense stacking positions and large steel plate storage volume, the present invention relies on hierarchical near-end strategy optimization to decouple the dual-layer coupled decision of target plate outbound and obstacle plate re-stacking, effectively solving the problems of long solution time and poor scheduling scheme quality of traditional iterative search algorithm; the dual-layer framework greatly reduces the computation cost of large-scale scheduling problems, can quickly generate real-time scheduling schemes, and improve the efficiency of steel plate logistics in the factory area.

[0037] (2) This invention takes into account both the cost of daily stacking operations and the risk of stack blockage after the scheduling cycle ends, and constructs a dual-objective collaborative optimization mechanism to incorporate the potential stacking hazards generated by steel plates to be picked up the next day into the overall optimization objectives. Compared with optimizing only the stacking operation scheduling on the same day, it can significantly reduce the total amount of subsequent stacking operations in the yard.

[0038] (3) The designed hierarchical near-end strategy optimization framework can complete the offline training and online deployment of the model based solely on the basic working condition data of the yard. It does not require additional yard detection hardware or customized special solution equipment. It has low deployment cost, strong generalization ability, and can be adapted to shipyard steel plate yard outbound scheduling scenarios of different scales and different steel plate outbound plans.

[0039] The advantages of this invention are as follows:

[0040] (1) Compared with the traditional shipyard steel plate yard scheduling scheme based on the exact solver, the present invention designs a hierarchical near-end strategy optimization framework for the large-scale yard coupled decision problem. It decouples the optimization of the target steel plate outbound order and the obstacle steel plate stacking into two-level decision-making. After offline training, it can be applied to the real-time scheduling on site without online repeated training, which greatly reduces the computation time of online scheduling.

[0041] (2) Compared with the single-level deep reinforcement learning steel plate scheduling method, this invention splits the original ultra-high-dimensional joint action space and designs hierarchical intelligent agents that are adapted to global outbound planning and local stacking allocation. Even in shipyard operation scenarios with dense stacks and a large number of steel plates, it can still stably output a scheduling scheme that takes into account both long and short cycle benefits and quickly complete the steel plate outbound decision.

[0042] (3) Compared with the yard scheduling algorithm that only takes the daily stacking volume as the optimization target, the present invention integrates the daily stacking cost and the stacking blockage risk after the scheduling is completed, incorporates the potential stacking risks of steel plates to be picked up the next day into the collaborative optimization system, and avoids the stacking blockage problem by relying on the forward perception capability of the hierarchical intelligent agent. Without increasing the single scheduling calculation overhead, the optimization effect of yard layout under continuous production conditions is effectively improved. Attached Figure Description

[0043] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0044] The technical solution of the present invention will now be described in detail with reference to the accompanying drawings.

[0045] This invention provides a shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning, comprising:

[0046] Collect basic operating data of the shipyard's steel plate stockpile and construct the original dataset for stockpile scheduling.

[0047] Based on basic working condition data, a mathematical model for the dispatching of steel plates out of the shipyard is established, including an objective function and constraints. The objective function includes a term that minimizes the total number of stacking operations on the day and a term that minimizes the degree of obstruction of the stacking position when the dispatching is terminated.

[0048] A hierarchical near-end strategy optimization scheduling framework is constructed. The framework includes a high-level agent and a low-level agent. The high-level agent is used to output the target steel plate outbound sequence, and the low-level agent is used to allocate the stacking position of the obstacle steel plates according to the target steel plate outbound sequence.

[0049] Design the state feature vectors, action spaces, and reward functions for high-level and low-level intelligent agents;

[0050] The high-level agent and the low-level agent are jointly trained offline based on the original data set of the yard scheduling to obtain the trained scheduling model.

[0051] Input real-time operating data into the trained scheduling model to output a steel plate outbound scheduling scheme.

[0052] The following is a detailed implementation process of the present invention.

[0053] like Figure 1 As shown, the present invention provides a shipyard steel plate storage and retrieval scheduling method based on hierarchical near-end strategy optimization, comprising the following steps:

[0054] Step 1: Collect basic operating data of the shipyard steel plate storage yard, including the storage yard capacity, the stacking order of steel plates in each storage yard, the list of steel plates to be shipped out on the day, the distribution of steel plates to be picked up the next day, the upper limit of stacking height, and other real-world operating information. Construct the original dataset for storage yard scheduling to provide data support for subsequent mathematical model construction and hierarchical reinforcement learning agent training.

[0055] Step 2: Establish a mathematical model for the dispatching of steel plates from the shipyard.

[0056] First, define the following parameters and decision variables.

[0057] (1) Parameters:

[0058] The collection of all steel plates in the storage yard. ;

[0059] : A collection of target steel plates that need to be shipped out on the same day;

[0060] : Collection of steel plates to be shipped out later;

[0061] : A collection of non-target steel plates for which there is currently no plan for shipment;

[0062] : Collection of storage yard stacking sites;

[0063] The maximum height that can be stacked in a stack, i.e. the maximum number of steel plates that can be stored in a single stack.

[0064] The set of discrete time steps throughout the entire scheduling period. This represents the time step before the scheduling ends.

[0065] A collection of slots within a single stack, with slot numbers arranged in ascending order from bottom to top, representing the maximum number of steel plates that can be stored in a single stack. Represents the maximum slot number;

[0066] : Initial state identifier of the storage yard; if steel plate i is initially stored in the c-th layer slot of stack position k, then ,otherwise .

[0067] (2) Decision variables:

[0068] If in time step If steel plate i is stored in the c-th slot of stack position k, the value is 1; otherwise, the value is 0.

[0069] If in time step If steel plate i is displaced from the c-th layer slot of stack position k, the value is 1; otherwise, the value is 0.

[0070] If in time step If steel plate i is moved to the c-th slot of stack position k, the value is 1; otherwise, the value is 0.

[0071] If steel plate i is at time step If the outbound operation is completed, the value is 1; otherwise, the value is 0.

[0072] If the scheduling terminates at the time step If a non-target steel plate is stored in the c-th layer of the stack position k, and a steel plate to be shipped out is present in any slot below the steel plate, then the value is 1; otherwise, the value is 0.

[0073] Based on the symbols defined above, the steel plate outbound scheduling problem can be expressed as the following mathematical model P1.

[0074] P1: (1) (2) (3)

[0075] The objective function of mathematical model P1 includes , Two items. This represents the total number of steel plate re-stacking operations during the entire scheduling cycle of the day. Minimizing this indicator can directly reduce the number of re-stacking operations in the stockyard on that day. Used for quantizing the termination time step of the scheduling The quantity of steel plates to be shipped out later is temporarily blocked. Minimizing the stacking can optimize the layout of the storage yard and effectively reduce the repacking cost of subsequent outbound tasks.

[0076] Model constraints are categorized and described according to their specific functions, as shown below.

[0077] 1) Constraints on the spatial layout and state evolution of the storage yard

[0078] (4) (5) (6) (7) (8) (9)

[0079] Constraint (4) Based on parameters The steel plate distribution is initialized, and constraint (5) stipulates that at most one steel plate can be placed in each slot of each stack within any time step. Constraint (6) stipulates that steel plates cannot be placed in the air. Constraint (7) limits the total number of steel plates in each stack to no more than the maximum allowable stack capacity. Constraint (8) stipulates that the removal operation can only be performed when a steel plate is present at the stack location. Constraint (9) is the state transition equation, based on the removal action variables. With the move-in action variable Update time step to The dynamic occupancy status.

[0080] 2) Operational logic constraints

[0081] (10) (11) (12) (13) (14) (15) (16) (17) (18) (19) (20)

[0082] Constraints (10) and (11) ensure that all target steel plates are dispatched exactly once, while strictly prohibiting dispatching operations on subsequent dispatched steel plates and steel plates without dispatch plans. Constraint (12) stipulates that target steel plates cannot be restashed, while constraint (13) restricts the system to perform dispatching or restashing actions at most once per time step. To ensure the rationality of the scheduling scheme, constraint (14) specifies that a steel plate can only be moved out when the slot above it is empty. Constraint (15) sets a timing termination logic: this constraint stipulates that if the yard crane is in time step Once in an idle state, it must remain completely idle for all subsequent time steps to avoid any redundant or invalid restacking operations after the operation is completed. Constraint (16) states that once a steel plate is removed, it must either be released from the warehouse or restashed to another stack. Constraint (17) eliminates operational conflicts caused by simultaneous steel plate entry and exit. Constraint (18) ensures that a steel plate will not reappear in any stack after it has been released from the warehouse. Constraint (19) stipulates that for any stack k that does not contain the target steel plate, only steel plate entry operations are allowed, and it is prohibited to remove any non-target steel plate from that stack. Constraint (20) stipulates that if the crane is in time step If a barrier steel plate is moved out of a certain stack, then in the adjacent time step Steel plates within the same stack must be processed continuously until the target steel plate is successfully removed from the warehouse.

[0083] 3) Obstruction level constraints

[0084] (21) (22) (23)

[0085] Constraints (21) to (23) establish the scheduling termination time step. The following quantification expression for obstruction level is used to assess the potential impact of yard layout on subsequent outbound operations. Specifically, this set of constraints is used to determine whether non-target steel plates are stacked on top of subsequent outbound steel plates at the end of the scheduling process; if so, it forces... The value is 1.

[0086] Step 3: Based on the MILP model established in Step 2, construct a hierarchical near-end policy optimization two-layer scheduling framework, including two independent Actor-Critic networks: a high-level agent and a low-level agent. The high-level agent is responsible for global yard state observation and outputs the approximate optimal target steel plate outbound sequence for the day; the low-level agent receives outbound tasks issued by the high-level agent and completes the stacking and allocation of stacking positions for steel plates with obstacles within the stack; the high-level and low-level agents adopt nested interaction logic, with a single high-level decision corresponding to a complete low-level stacking sub-task, until the target steel plate is exposed on top of the stack and the outbound process is completed.

[0087] Step 4: Design the state features, actions, and dual-objective driven reward functions for high-level and low-level agents respectively.

[0088] Step 5: Construct an Actor-Critic network with layer normalization and GELU activation, and complete offline joint training of high- and low-level agents based on nested interactive sampling;

[0089] Step Six: Load the trained network and input the real-time operating conditions of the yard to achieve online scheduling and solution in seconds;

[0090] Step 7: Output a steel plate storage and retrieval plan that takes into account both the current stacking volume and the risk of future blockage, to guide on-site yard operations.

[0091] The invention will be further illustrated below with specific examples from industrial scenarios:

[0092] A hierarchical near-end strategy optimization-based method for scheduling the storage and retrieval of steel plates in shipyards, the main implementation steps of which are as follows:

[0093] Step 1: Collect 180 case studies of different sizes. The data format of each case study includes steel plate serial number, stack number, layer number, and steel plate outbound priority, as shown in Table 1. In the "Steel Plate Outbound Priority" column, 1 represents the target steel plate that needs to be outbound on the current day, 2 represents the steel plate to be outbound later, and 3 represents non-target steel plates that do not currently have an outbound plan.

[0094] Table 1

[0095]

[0096] Step 2: Establish a mathematical model P1 for the dispatching of steel plates from the shipyard.

[0097] Step 3: Construct a hierarchical near-end strategy optimization two-layer intelligent scheduling framework to decouple and separate the coupled outbound and stacking decisions in the mathematical model of Step 2. The two-layer scheduling framework includes a high-level global outbound planning agent and a low-level stacking obstacle plate stacking allocation agent. Both agents are implemented using the PPO algorithm, and each layer is equipped with an independent Actor-Critic network and experience replay pool. The high-level agent is responsible for observing the global state of the yard and outputting the stacking position number of the target steel plate to be outbound at the current stage. The low-level agent receives the outbound task issued by the high-level agent and completes the stacking position allocation of the obstacle steel plates in the stack. The high- and low-level tasks are linked through nested interaction logic. A single high-level outbound task corresponds to a complete low-level relocation sub-process until all the obstacle steel plates at the top of the stack are removed and the target steel plate can be directly outbound.

[0098] Step 4: Design of state characteristics, actions, and reward functions for high-level intelligent agents.

[0099] (1) High-level agent state design: The state vector of the high-level agent is formed by concatenating the normalized features of all stack positions:

[0100] a. Percentage of unplanned steel plate blockages ( If stack k contains target outbound steel plates in high-level decision step T, then this feature is the ratio of the number of steel plates without outbound plans in the stack to the maximum storage capacity of the stack; in other scenarios, this feature is 1.

[0101]

[0102] in, This indicates the number of steel plates without a planned outbound shipment in stack position k, which is obscured by the high-level decision-making step T.

[0103] b. Percentage of obstructed steel plates to be retrieved If stack position k contains the target steel plate to be shipped out in the high-level decision step T, then the value of this feature is the ratio of the number of subsequent steel plates to be shipped out that are blocking the target steel plate in the stack to the maximum storage capacity of the stack position; in other scenarios, the value of this feature is 1.

[0104]

[0105] in, This indicates the number of steel plates to be shipped out from the warehouse in the stack position k, which is the target steel plate that is blocked by the high-level decision-making step T.

[0106] c. Stacking position release degree ( ): Used to quantify the layout quality of stack position k after the top target steel plate is removed.

[0107]

[0108] Where x is the number of steel plates remaining in the stack after the top steel plate is removed; the coefficient a is determined according to the following rules: 1) when the remaining plates include the target steel plates to be shipped out on the same day, a=0; 2) when the remaining plates only contain steel plates to be shipped out later and no target steel plates to be shipped out on the same day, a=0.5; 3) when the remaining plates contain neither target steel plates to be shipped out on the same day nor steel plates to be shipped out later, a=1.

[0109] For any stack position Extracting the 3D stack feature vector ,in This represents the maximum number of stack positions contained in all the training examples. When the actual number of stack positions in a training example is less than the maximum number of stack positions... At this time, zero-padding is performed on the insufficient feature vectors to ensure that the input dimension of the neural network remains constant under different scale problems. At the same time, an action masking mechanism is introduced to prevent the agent from selecting invalid stack positions. Finally, the complete high-level state is represented as: .

[0110] (2) High-level agent action design: The high-level action is to select the near-optimal target stack location for outbound shipment. Specifically, in each high-level decision step T, the high-level agent will select a target stack location. The goal is to retrieve the topmost target steel plate and its adjacent target steel plates from the stack. To eliminate invalid exploration behavior and ensure that the decision-making logic conforms to the on-site operation rules, an action masking mechanism is introduced at the output layer of the policy network to avoid selecting infeasible stacks.

[0111] (3) High-level agent reward design: If there is an obstacle steel plate above the target steel plate, then a single decision step T of the high-level agent corresponds to a complete stacking operation sequence. Therefore, the high-level agent's reward... Defined as the average of all immediate rewards for lower-level agents, this avoids gradient manipulation by tasks with excessively long shift sequences. Furthermore, if the number of obstructing steel plates above the target steel plate is 0, the higher-level agent will receive a reward of 1, thus guiding the agent to prioritize stack positions where a shift operation is not required in the current decision step. Higher-level rewards The expression is as follows:

[0112]

[0113] in, The total number of obstacle steel plates that need to be stacked in step T is the decision step.

[0114] Step 5: Design the state characteristics, actions, and reward functions of the low-level agent.

[0115] (1) Low-level agent state design: The low-level agent state vector is composed as follows:

[0116] a. Inverted steel plate markings ( ): Used to distinguish the type of the obstacle steel plate to be overturned at the stack position at low-level decision step t, defined as follows:

[0117]

[0118] b. Stack space utilization rate ): Characterizes the stack position k relative to the maximum stack height. The normalized occupancy level. At each low-level decision step t, These will be dynamically updated to reflect the real-time capacity changes of the stacking positions during the stacking operation, as defined below:

[0119]

[0120] in, This represents the total number of existing steel plates in position k at the lower-level decision step t.

[0121] c. Blocking strength ( This feature divides the total number of obstacle steel plates within stack position k at the lower-level decision step t by the maximum capacity of the stack position. After normalization, the congestion rate of the candidate pallet positions is calculated. This feature provides the low-level agent with reference information on the potential penalty cost of future outbound operations for each pallet position, and is defined as follows:

[0122]

[0123] in, This represents the total number of obstacle steel plates within stack position k at the time of the lower-level decision step t.

[0124] d. Degree of impact of single-step stacking ( ): By virtually placing the top layer of the barrier steel plate at the target stack position j Reposition the stack to candidate stack position k, evaluate the impact of this relocation operation on subsequent outbound efficiency, and enable the low-level agent to have forward-looking predictive capabilities, defined as follows:

[0125]

[0126] in Represents the highest outbound priority number; , Steel plates to be stacked Priority number, minimum priority number of steel plate in stack position k.

[0127] e. Extended stacking effect ( The feature extends the impact of the stacking to the next obstacle steel plate to be stacked. Compared to the influence characteristics of single-step shift It enables intelligent agents to predict the risk of cascading blockages that may occur during continuous stacking operations, as defined below:

[0128]

[0129] In summary, the low-level state vector is composed of global features. It is formed by concatenating the local feature vectors of all stack locations in the storage yard, i.e. If the actual total number of stack positions in a given calculation is less than the maximum number of stack positions. Then, zero-padding is performed on the feature vector to ensure that the input dimension of the neural network remains constant.

[0130] (2) Low-level agent action design: The low-level action space is defined as a set of positions. any action The selected target stack position k is used to store the currently unpiled steel plates. To ensure the feasibility of the action, an action masking mechanism is also used to block invalid stack positions.

[0131] (3) Low-level agent reward design: If there is no steel plate with an earlier outbound time under the reversed steel plate after the stacking is reversed, a positive reward is given. Conversely, if the stacked steel plates are stored on top of steel plates that were issued earlier, a negative reward is applied. .

[0132] Step 5: Utilize the designed hierarchical near-end strategy to optimize the two-layer intelligent scheduling framework for algorithm training. The training process is as follows: Figure 1 As shown.

[0133] Step 6: Perform real-time scheduling based on the trained network. The scheduling results of the example described in Table 1 are shown in Table 2.

[0134] Table 2

[0135]

[0136] The above are preferred embodiments of the present invention. Any changes made to the technical solution of the present invention that do not exceed the scope of the technical solution of the present invention shall fall within the protection scope of the present invention.

Claims

1. A shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning, characterized in that, include: Collect basic operating data of the shipyard's steel plate stockpile and construct the original dataset for stockpile scheduling. Based on basic working condition data, a mathematical model for the dispatching of steel plates out of the shipyard is established, including an objective function and constraints. The objective function includes a term that minimizes the total number of stacking operations on the day and a term that minimizes the degree of obstruction of the stacking position when the dispatching is terminated. A hierarchical near-end strategy optimization scheduling framework is constructed. The framework includes a high-level agent and a low-level agent. The high-level agent is used to output the target steel plate outbound sequence, and the low-level agent is used to allocate the stacking position of the obstacle steel plates according to the target steel plate outbound sequence. Design the state feature vectors, action spaces, and reward functions for high-level and low-level intelligent agents; The high-level agent and the low-level agent are jointly trained offline based on the original data set of the yard scheduling to obtain the trained scheduling model. Input real-time operating data into the trained scheduling model to output a steel plate outbound scheduling scheme.

2. The shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The basic operating data includes: the capacity of the storage yard, the stacking order of steel plates in each storage yard, the list of steel plates to be shipped out on the current day, the distribution of steel plates to be picked up the next day, and the upper limit of the stacking height.

3. The shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The decision variables in the shipyard steel plate outbound scheduling mathematical model include: the time occupancy state variable of steel plates in the stacking slots. Variables of steel plate removal action Variables for steel plate movement 1. Steel plate outbound completion status variable and blocking indicator variables ;in, Let i, k, and c be the time steps, representing the i-th steel plate, the k-th stack position, and the c-th layer slot position, respectively. The constraints include constraints on the spatial layout and state evolution of the storage yard, operational logic constraints, and obstruction level constraints.

4. The shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The hierarchical near-end strategy optimization scheduling framework adopts nested interaction logic. A single decision by a high-level agent corresponds to a round of stack-turning sub-tasks by a low-level agent, until the target steel plate is exposed to the top of the stack and the outbound process is completed.

5. A shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The state feature vector of the high-level intelligent agent is composed of the three-dimensional features of all stack positions, and the three-dimensional features include: the proportion of unplanned steel plate blockage. The proportion of steel plates blocked before being retrieved and stack release degree ; The action space of the high-level intelligent agent is to select the target stack number to be shipped out. ; The reward function of the high-level intelligent agent Defined as: in, To determine the total number of obstacle steel plates that need to be re-stashed in step T, This is the immediate reward for the lower-level intelligent agent at time t.

6. A shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning according to claim 5, characterized in that, The stack release degree The calculation method is as follows: Where x is the number of steel plates remaining in the stack after the top steel plate is removed; the coefficient a is determined according to the following rules: 1) when the remaining plates include the target steel plates to be shipped out on the same day, a=0; 2) when the remaining plates only contain steel plates to be shipped out later and no target steel plates to be shipped out on the same day, a=0.5; 3) when the remaining plates contain neither target steel plates to be shipped out on the same day nor steel plates to be shipped out later, a=1.

7. A shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The state feature vector of the low-level agent includes the identifier of the stacked steel plate. Stack space utilization rate Blockage intensity The degree of impact of single-step stacking Extended stacking impact ; The action space of the lower-level intelligent agent is to select the candidate stacking position number for storing the current obstacle steel plate to be overturned. .

8. A shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning according to claim 7, characterized in that, The impact of single-step stacking The calculation method is as follows: in Represents the highest outbound priority number; , Steel plates to be stacked The priority number and the minimum priority number of the steel plate in stack position k, where k and j are the candidate stack position and the target stack position, respectively.

9. A shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, The reward function of the low-level agent is configured as follows: if, after the stacking, there is no steel plate with an earlier outbound time below the stacked steel plate, a positive reward is given. ; If the stacked steel plates are stored on top of steel plates that were issued earlier, a negative reward will be given. .

10. A shipyard steel plate outbound scheduling method based on hierarchical reinforcement learning according to claim 1, characterized in that, An action masking mechanism is introduced in the policy network output layer of both high-level and low-level agents to filter out invalid stacking position selection actions that do not conform to the on-site operation rules. When the actual number of stack positions in the calculation is less than the preset maximum number of stack positions At that time, zero-padding is performed on the insufficient parts of the state feature vectors of high-level and low-level intelligent agents to fix the input dimension of the neural network.

Images