Automatic port import container berth allocation method embedded with deep reinforcement learning

Abstract

The application discloses an automatic wharf import container bay allocation method embedded with deep reinforcement learning. The method comprises the following steps: firstly, for the container picking area, a target function is established based on the total processing time of the batch container and the total time of turning over all future containers; then, the established mathematical model is subjected to robust equivalence conversion, the uncertainty model is converted into a deterministic model, and a constraint of the bay layout selection is newly added; subsequently, the possible layout of the bay is counted and simplified, so that the number of constraint conditions in the model is greatly reduced; the upper limit value of the container turning in the target function after the robust equivalence conversion is solved by the deep reinforcement learning method; and finally, a container allocation scheme is obtained by using a commercial solver. On the basis of significantly improving the solution efficiency of the container allocation strategy, the application ensures that the wharf has higher stability and reliability when facing the import container operation with high uncertainty, and provides strong decision support for the wharf operation management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of container scheduling, and more particularly to an automatic wharf import container bay allocation method embedded with deep reinforcement learning. BACKGROUND

[0002] With the continuous development of global supply chain, major ports have upgraded their container terminals to improve port efficiency. The rise of new automated container terminals marks the entry of port automation into a new stage. One of the notable features of these new automated terminals is the change in gantry equipment. New automated terminals mainly use single-side cantilever rail gantry cranes (URMGC). Compared with the automatic stacking cranes (ASC) widely used in traditional automated terminals, the cantilever part of URMGC extends to one side of the crane, increasing the lateral movement range of the crane. The introduction of this equipment has brought significant changes to the layout of the yard. As shown in Figure 1 , in the yard layout of new automated terminals, the passageway between adjacent block areas is open to automated guided vehicles (AGV), i.e., AGV can drive into the yard area and interact with the crane below the cantilever protruding end. In traditional automated terminals, AGV can only interact with the crane at the end of the yard (water side). The significant change in yard layout has led to a fundamental change in yard operation mode. Taking the import container scenario of Figure 1 , for example, in new automated terminals, containers enter the yard from the water side mainly relying on automated guided vehicles, which can significantly reduce the travel of gantry cranes. In traditional automated terminals, containers enter the yard can only rely on the longitudinal movement of gantry cranes along the guide rail from the water side to the land side. Since the moving speed of gantry cranes is much lower than that of AGV, it means that new automated terminals have higher single-container handling efficiency.

[0003] Import container bay allocation is an important decision that port enterprises need to make in terminal operations. Reasonable bay allocation can effectively help port enterprises to reduce costs and improve terminal efficiency, which is of great significance to alleviate the long waiting time of ships, port congestion and other key issues that hinder the convenience of global trade. Specifically, the import container bay allocation problem refers to allocating a usable bay for each arriving container at the port within a certain period of time, based on the current characteristics of the equipment, layout features, and existing bay resources of the yard, and considering uncertain factors such as future container pickup time and pickup order, so as to minimize the total operation time (time cost) for a given number of import containers. Terminal operations usually include two stages. The first stage is the stacking operation, i.e., stacking the given number of import containers to the specified bay. The second stage is the picking operation, i.e., picking the specified containers from the yard for the consignee to pick up the goods.

[0004] Existing technologies are mainly oriented towards traditional automated terminal scenarios, with the goal of minimizing terminal operation time (or time cost). First, a mathematical programming model that takes uncertainty into account is constructed for the allocation of import container slots, and then a solution method is designed for the model. In terms of mathematical models, the existing technology's characterization of the stacking phase is based on the characteristics of the interaction between the AGV and the yard crane on the water side of the traditional automated terminal; the characterization of the container picking phase mainly considers the additional container turning operations caused by the uncertainty of the container picking information. In terms of solution methods, existing technologies mainly solve the model based on stochastic programming methods. For example, the probability distribution of the relevant parameters of the container picking information is first fitted based on historical data, and then multiple container picking scenarios are generated based on the probability distribution. The container slot allocation problem that takes uncertainty into account is transformed into a deterministic model with the goal of minimizing the expectation of terminal operation time under multiple scenarios. Finally, a commercial solver or an exact algorithm is used to solve it.

[0005] Analysis has shown that existing technologies are not suitable for modern automated terminals. This is because the key feature of modern automated terminals in handling imported containers is the interaction between AGVs and yard cranes in the yard aisles. However, existing mathematical models, based on traditional automated terminal scenarios, fail to capture these interactive characteristics. Furthermore, stochastic optimization methods, when solving uncertainty-based slot allocation models, primarily employ multi-scenario expectation planning to address the uncertainty of the container retrieval phase. However, in practice, the slot allocation decisions provided by these methods have failed to achieve the desired effect in reducing the number of additional container transfers during the container pickup phase. This is due to the following reasons: First, the high uncertainty of container retrieval information (e.g., retrieval time and order) makes it impossible to accurately reflect the statistical characteristics of the overall sample of container retrieval information. This means that in actual operations, there are a large number of scenarios that cannot be simulated by existing technologies. Second, using the expected terminal operation time under multiple scenarios as the optimization objective essentially makes slot allocation decisions based on probability-weighted scenarios. This results in excessive sensitivity of the number of container transfers to changes in retrieval information. Specifically, if the actual container retrieval information matches a scenario with a higher probability weight, that is, a more frequent container retrieval scenario, then the container slot allocation scheme provided by the existing technology can provide better feedback (fewer container tumbles during retrieval). If the actual container retrieval information matches a scenario with a lower probability weight, that is, a less frequent container retrieval scenario, then the container slot allocation scheme provided by the existing technology may not reduce the number of container tumbles, and may even increase the number of container tumbles. Summary of the Invention

[0006] The present invention aims to overcome the shortcomings of the aforementioned prior art by providing a method for allocating import container slots at automated terminals embedded in deep reinforcement learning. This method includes: establishing an objective function for the container pickup area based on the total processing time of the current batch of containers and the total container turnover time required to remove all containers in the future; performing a robust equivalent transformation on this objective function to obtain a simplified deterministic objective function; and solving this simplified objective function using reinforcement learning to obtain a slot allocation solution.

[0007] Compared with the prior art, the application has the advantages that a brand-new container bay allocation model is constructed, the operation characteristics of the single cantilever rail-mounted gantry crane (URMGC) and the automatic guided vehicle (AGV) are deeply considered, and the change of the new yard layout is considered, so as to ensure that the container bay allocation decision is highly consistent with the actual operation environment, and the operation efficiency and resource utilization of the terminal are significantly improved. In addition, the method provided by the application is based on robust optimization, and the container bay allocation scheme is decided, and the extremely complex container turning conditions caused by the flexible and changeable container taking sequence are fully considered, so as to provide time (time cost) redundancy for the actual terminal operation, and ensure that the terminal efficiency has higher stability and reliability in the face of the import container operation with high uncertainty under the decision scheme.

[0008] Other features and advantages of the present application will become apparent from the following detailed description of illustrative embodiments thereof, which proceeds with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0009] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0010] Figure 1 is a schematic diagram of different automated container terminals of the prior art;

[0011] Figure 2 is a flowchart of an automated container terminal import container bay allocation method embedded with deep reinforcement learning according to an embodiment of the application;

[0012] Figure 3 is a schematic diagram of a container yard according to an embodiment of the application;

[0013] Figure 4 is a schematic diagram of a new automated container terminal layout according to an embodiment of the application;

[0014] Figure 5 is a schematic diagram of a container turning operation according to an embodiment of the application;

[0015] Figure 6 is a three-dimensional schematic diagram of a new automated container terminal yard operation according to an embodiment of the application;

[0016] Figure 7 is a flowchart of an upper bound solution of a container turning operation according to an embodiment of the application;

[0017] Figure 8 is a schematic diagram of a new automated container terminal bay allocation solution framework according to an embodiment of the application. DETAILED DESCRIPTION

[0018] Various exemplary embodiments of the present application will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions, and numerical values set forth in these embodiments are not limitations on the scope of the present application unless otherwise specifically indicated.

[0019] The following description of at least one exemplary embodiment is merely exemplary in nature and is in no way intended to limit the scope of the application its application or uses.

[0020] Techniques, methods, and devices known to those of ordinary skill in the relevant art can not be discussed in detail herein. However, where appropriate, such techniques, methods, and devices can be viewed as part of the specification.

[0021] In all of the examples shown and discussed herein, any specific values should be interpreted as merely exemplary, and not as a limitation. Thus, other examples of exemplary embodiments can have different values.

[0022] It should be noted that like reference numerals and letters in the various figures indicate similar items, and thus, once any certain item is defined in one figure, it is not necessary to discuss it further in subsequent figures.

[0023] The present application proposes an innovative import box bay allocation method. The method mainly includes: a box bay allocation model that adapts to the layout and equipment interaction characteristics of a new type of automated terminal; a model solving method based on the robust optimization idea and integrating deep reinforcement learning technology. The box bay allocation method aims to solve the following problems:

[0024] 1) Considering the defects of the existing box bay allocation model that cannot adapt to the new type of automated terminal, a new import box bay allocation model considering uncertainty is constructed. The model aims to minimize the operation time of the new type of automated terminal, fully considers the yard layout of the new type of automated terminal, the cantilever operation of the URMGC, and the movement characteristics of the AGV in the yard, can accurately depict the yard layout of the new type of automated terminal and the characteristics of the interaction between the gantry crane and the AGV in the yard, and ensures that the model can accurately calculate and optimize the operation time of the new type of terminal in the stacking stage.

[0025] 2) A model solving method based on the robust optimization idea and integrating deep reinforcement learning technology is designed. Since the existing technology decision box bay allocation scheme has the problem of high sensitivity of the number of box turning operations in the box picking operation to the change of the box picking information, the present application designs a model solving method for the box bay allocation model considering uncertainty based on the robust optimization idea, considers the maximum limit of the number of box turning operations due to the uncertainty of the box picking sequence. Ensures that the terminal operation time under this limit condition can still be optimized, thereby reducing the high sensitivity of the box turning operation to the change of the box picking sequence.

[0026] In general, the method for automated import container slot allocation at a terminal embedded with deep reinforcement learning provided by the present invention includes the following aspects:

[0027] 1) Given a container stacking layout (all bays are identical) within the yard, find the order (i.e., the retrieval sequence) that results in the highest number of container turnovers under that stacking layout. This retrieval sequence is defined as the worst retrieval sequence, and the corresponding number of container turnovers is the upper bound of the number of container turnovers for that stacking layout.

[0028] 2) Using the mapping relationship between stacking layouts and the upper bound on the number of container turnovers, constraints are established for all possible stacking layouts. Since there are numerous possible stacking layouts, the number of container turnovers must not exceed the upper bound if a given stacking layout is used. This constraint ensures that the maximum number of container turnovers will not be exceeded regardless of the order in which the containers are retrieved, eliminating uncertainty in the order in which the containers are retrieved. Therefore, this constraint can transform a model that considers uncertainty into a deterministic model.

[0029] 3) Solve the deterministic model using an exact or heuristic optimization algorithm to ultimately determine the bin allocation solution. Due to the extremely large number of combinations of "stack layout - bin retrieval order," iterating through all combinations to determine the worst-case retrieval order and an upper bound on the number of bin turnovers would result in the curse of dimensionality and be nearly impossible. Therefore, deep reinforcement learning techniques are introduced. Based on finite-dimensional data, a model is trained to capture the "stack layout - worst-case retrieval order" mapping. This model is then used to determine an upper bound on the number of bin turnovers for a given stack layout.

[0030] For example, based on the Actor-Critic framework, an Actor neural network generates actions or action probability distributions (such as swaps, reversals, and insertions) in the current state to optimize the search order strategy. Simultaneously, a Critic neural network evaluates the value (V) of the current state and action pair. The Actor and Critic networks are collaboratively trained using the calculated TD Error (Time Difference Error), gradually adjusting parameters until the TD Error converges, thereby optimizing the strategy and approaching the global optimal solution.

[0031] For example, a custom compact upper bound evaluation module is introduced to accelerate the training of Actor-Critic. This module can provide timely reward evaluation for Actor-Critic by quickly calculating the difference in the compact upper bound of the number of container flipping between the current state and the new state. In addition, when conducting multi-layout collaborative training for the same number of containers, the module quickly matches the pre-stored layout information through the layout identifier and independently generates reward values ​​for specific layouts. This method significantly accelerates the training convergence process and shortens the overall time required to implement the strategy. When the training reaches the termination condition, the final retrieval order will be input into the precise container flipping value calculation module to ensure the accuracy of the container allocation model during the solution process and to ensure that the resulting solution is highly reliable in practical applications.

[0032] Specifically, see Figure 2 As shown, the provided automated terminal import container slot allocation method embedded with deep reinforcement learning includes the following steps:

[0033] Step S110 , constructing a mathematical model for slot allocation at a new automated container terminal, and establishing an objective function based on the total processing time of the current batch of containers and the total container turnover time required to remove all containers in the future.

[0034] 1. Problem description

[0035] In the new automated container terminal, the yard provides a temporary storage place for a large number of containers that need to be circulated in the terminal. The yard is composed of block areas, and the containers are stored in a specific position in the block area, such as Figure 3 shown.

[0036] In traditional automated container terminals, automated guided vehicles (AGVs) are only responsible for transporting containers to the docking areas on both sides of the container area, and there are no roads inside the container area for AGVs to move. The vertical movement of containers assigned to specific bays and the horizontal movement between stacks are all completed by yard cranes. In new automated terminals, AGVs can already penetrate deep into the container area to achieve precise control and positioning. When AGVs perform container storage tasks, they are responsible for moving containers to specific bays, while the movement of containers between stacks is undertaken by yard cranes. Using faster-moving AGVs to replace yard cranes for movement between bays can effectively improve the overall operational efficiency of the terminal, see Figure 4 The floor plan of the new automated container terminal is shown.

[0037] After an import container is unloaded from a ship by a quay crane, an automated guided vehicle (AGV) transports it from the coast to a designated bay. A yard crane then unloads the container from the truck and temporarily stores it in a specific stack in the container area, awaiting pickup by an outbound truck. When the container is ready for pickup, the yard crane transfers the container from its storage location to an outbound truck for transport to the inland hinterland.

[0038] When a yard crane unloads an imported container from an internal container truck and temporarily stores it in a container area, the yard operator must select the most appropriate storage location for the container among all available container slots in the area. This decision-making process of selecting the most appropriate storage location for an imported container is known as the import container slot allocation problem.

[0039] The rationality of slot allocation for large import containers not only directly impacts the time liner vessels spend waiting to unload but also indirectly influences the time future container trucks spend in the container transfer area. This is because future container pickups are handled by highly flexible container trucks, rather than more predictable vessels. Since the terminal cannot determine the arrival order of container trucks, the order in which import containers will be picked up cannot be predicted in advance. This can result in the target container being trapped beneath other containers during pickup. Upon reaching the transfer area, the container truck will need to wait for the yard crane to turn the container over before retrieving the target container.

[0040] Container turnover means that when the target container is covered by other containers (blocking containers), the yard crane must first move the blocking container above to a stack with vacant container space before successfully retrieving the target container. Figure 5 As shown in the figure, assume that the target container for the current foreign container truck to pick up is box C1 located in stack 3, and box C1 is located below boxes C2, C3, and C4. The yard crane has to first move box C4 at the top of stack 3 to stack 2 (the first container turning operation), then move box C3 to stack 4 (the second container turning operation), and finally move box C2 to stack 4 (the third container turning operation), before it can successfully take out box C and hand it over to the foreign container truck. Obviously, the container turning operation will increase the total time consumed in the container picking operation, extend the waiting time of the container truck in the handover area, and reduce the work efficiency and service level of the terminal.

[0041] For terminal operators, when container pickup information is highly uncertain, they need to rationally consider the allocation of import container slots, which is the key to improving the terminal's container turnover efficiency, resolving operational efficiency bottlenecks, and improving customer service levels and satisfaction.

[0042] 2. Mathematical model for import container slot allocation

[0043] Basic assumptions

[0044] 1) Containers of the same batch will be assigned to the same container area.

[0045] 2) The number of AGVs assigned to a container area is limited, which is set to one in this model.

[0046] 3) Container storage and container pickup operations will not be carried out at the same time. Container pickup operations will only begin after the container space allocation is completed.

[0047] 4) Container types are homogeneous.

[0048] 5) The order in which each import container arrives at the port is known in advance.

[0049] index

[0050] i: Import container index, i∈{1,2,…,I}. Import containers are numbered in the order of arrival.

[0051] j: stack position index, j∈{1,2,…,J}. Stack position 1 is on the left side of the bin, and stack position j is on the right side of the bin.

[0052] k: Bay index, k∈{1,2,…,K}. Bay 1 is closer to the land side, and bay k is closer to the water side.

[0053] parameter

[0054] e1: The time it takes for the AGV to travel to the yard after loading the container from the shore crane.

[0055] e2: The time it takes for the AGV to return from the container yard to the quay crane and wait for the container to be delivered.

[0056] n1: Unit moving time of the AGV equipped with the container.

[0057] n2: Unit moving time of the empty AGV.

[0058] m1: Unit horizontal movement time of URMGC between stack locations.

[0059] m2: Unit horizontal movement time of URMGC between bays.

[0060] H: Floor height.

[0061] C: capacity in degrees Celsius.

[0062] θ jk : The initial number of containers in stack position j, stack position k. j∈{1,2,…,J}, k∈{1,2,…,K}.

[0063] r: The unit time of the box turning operation.

[0064] Decision variables

[0065] x ijk :If the import box i is assigned to stack position j, shell position k, then x ijk = 1. Otherwise, x ijk = 0. Where, i∈{1,2,…,I}, j∈{1,2,…,J}, k∈{1,2,…,K}.

[0066] Dependent variable

[0067] Si : Import box i is assigned to a stack position, i∈{1,2,…,I}.

[0068] L i : Import bin i is assigned to a bin, i∈{1,2,…,I}.

[0069] T i : The total processing time of imported container i, that is, from the time container i waits for the AGV to pick up the container at the quay crane to the time the AGV hands the container over to the URMGC. i∈{1,2,…,I}.

[0070] R(θ jk ;x ijk ):After the slots are allocated, the layout of the current bay k corresponds to the upper bound of the number of box turns required for future box pickup.

[0071] Objective function

[0072] In one embodiment, the objective function is set as:

[0073]

[0074] The objective function describes the operational capacity limit of the container area when processing the current liner. It consists of two objectives: one is the total processing time of the current batch of containers, and the other is the total container turnover time required to remove all containers in the future.

[0075] is about the decision variable x ijk In practice, for the same number of containers in a bay, the number of container rummages can vary significantly depending on the layout. For example, if six containers are stacked flat in a six-layer bay, no rummages will occur. However, if all six containers are placed in the same stack, rummages will occur, except when the retrieval order is bottom-up.

[0076] Therefore, in view of the fundamental difference in the number of box turning caused by this layout, the present invention improves the box turning function. That is, one layout corresponds to one number of box turning.

[0077] Constraints

[0078]

[0079] T1=(K-L1)n1+e1 (5)

[0080]

[0081]

[0082] Constraint (2) means that any container in the same batch must be allocated to a container area;

[0083] Constraints (3) and (4) are used to calculate the stack and bay locations to which each container is assigned;

[0084] Constraint (5) indicates that for the first container, after determining its assigned location, the gantry crane will wait at the bay in advance. At this time, the time it takes for the AGV to enter the aisle after loading the container on the water side and drive to the specific bay to hand it over to the gantry crane is the operation time of the first container;

[0085] Constraint (6) indicates that for subsequent containers, after the specific location to be assigned is determined, its operation time will be the maximum between the operation times of the field crane and the AGV. The field crane operation time is the time it takes to move from the location where the previous container is placed to the location where the current container will be assigned. The AGV operation time is the time it takes to move from the location where the previous container was transferred to the waterside, load the current container, and then drive to the designated location.

[0086] Constraint (7) indicates that the stacking of containers cannot exceed the layer height;

[0087] Constraint (8) indicates that the number of containers in the same bay cannot be too large. This is because when taking containers in the future, there may be a container turning operation, so a certain amount of space needs to be reserved.

[0088] Constraint (9) defines a 0-1 decision variable regarding the slot allocation.

[0089] In step S120 , robust equivalent transformation is performed on the established objective function to obtain a deterministic mathematical model, and corresponding constraints on the selection of bay layout are added.

[0090] In mathematical models, the box flipping function Represented as a layout, corresponding to a number of box turning.

[0091] However, for imported containers, since the container retrieval task is completed by flexible external trucks, the terminal cannot determine the arrival order of these trucks, and therefore cannot predict the order of retrieval of imported containers. Although the layout is determined, the uncertainty of the retrieval order makes It is still impossible to directly calculate the corresponding number of box turning times.

[0092] In one embodiment, a robust equivalence transformation is employed to transform this uncertainty model into a solvable deterministic model by determining the worst-case search order and calculating the corresponding number of container turnovers for each specific bay layout. This model is then further evaluated to assess the operational capacity limits of the container area when handling the current liner container load, taking into account the worst-case scenario. The robust equivalence transformation process is as follows:

[0093] The objective function of the original model [OP] can be simplified as:

[0094]

[0095] By using a deterministic approach, we consider the worst possible search order under a fixed layout within the bay:

[0096] R(·)~layout→U R (layout)

[0097] Among them, U R (·) represents the upper bound of the box flipping, that is, the box flipping value corresponding to the worst search order. For the n layouts contained in the bay, the upper bound of the box flipping can be found:

[0098] R(l1)≤U R (l1)

[0099] R(l2)≤U R (l2)

[0100] …

[0101] R(l n )≤U R (l n )

[0102] For the box flipping function R(·) with uncertainty, according to the definition of robust optimization:

[0103]

[0104] Among them, l is the decision variable, that is, the decision variable x in the box allocation decision ijk and parameter θ jk The layout is defined in the feasible domain L; s is the uncertainty variable, that is, the search order, which is defined in the set of all possible search orders S. s∈S R(l,s) takes the maximum value, which is the upper bound of the bin flipping, removing the uncertainty caused by the search order:

[0105]

[0106] For the original mathematical model, new parameters and decision variables are introduced to characterize the process.

[0107] Adding an index

[0108] τ: The possible layout index in the bay, τ∈{1,2,…,σ}. σ is the sum of all possible layouts.

[0109] New parameters

[0110] ω τ : symbolic values ​​of different layouts τ, τ∈{1,2,…,σ}.

[0111] Upper bounds of box flipping for different layouts τ, τ∈{1,2,…,σ}.

[0112] Adding new decision variables

[0113] y τk : If the layout in bay k is τ, then y τk = 1. Otherwise, y τk = 0. Where k∈{1,2,…,K}, τ∈{1,2,…,σ}.

[0114] The robust equivalence problem of the original model is:

[0115]

[0116] Add new constraints

[0117]

[0118] The objective function (10) takes into account the optimization of the container allocation for a batch of imported containers under the most unfavorable retrieval conditions in the future for the current container area. The value obtained is the maximum operating capacity of the container area under the current batch.

[0119] Constraints (11) and (12) together indicate that among all possible layouts, Beiwei can only choose one of them.

[0120] Constraints (13) define 0-1 decision variables regarding layout selection.

[0121] Step S130 , statistics and simplification of possible layouts of bays are performed to significantly reduce the number of constraints in the mathematical model.

[0122] For the possible layouts τ in a bay, we need to count all possible layouts σ and represent them appropriately. For example, a bay typically has six stacks with a height of six. For this example, we count all possible layouts for a bay with 1-25 containers, as shown in Table 1.

[0123] Table 1: Bay layout statistics

[0124]

[0125] As shown in the table, for a bay with six stacking slots and a height of six, there are approximately 100,000 possible layouts when the bay capacity is 25, taking into account both the box turning operation and safety factors. In the mathematical model, if the search order is not considered, this would introduce approximately 100,000 constraints, making the model construction and solution extremely difficult. In one embodiment, this problem is solved by summarizing and simplifying the layout.

[0126] For example, when there is only one container in a bay, there are six possible layouts: (1,0,0,0,0,0), (0,1,0,0,0,0), (0,0,1,0,0,0), (0,0,0,1,0,0), (0,0,0,1,0,0), (0,0,0,0,1,0), and (0,0,0,0,0,1). These layouts share the same elements (i.e., they all contain five zeros and one one), differing only in their order. They are therefore called "homogeneous layouts." These layouts are grouped as homogeneous, allowing them to share the same mapping between search order and container turnover count.

[0127] On the other hand, Figure 6 As shown in the figure, in a modern automated terminal container area, assuming the URMGC and AGV place a container at a designated bay (2,2,1,1,0,0), they must choose between the following layouts: (3,2,1,1,0,0), (2,3,1,1,0,0), (2,2,2,1,0,0), (2,2,1,2,0,0), (2,2,1,1,1,0), and (2,2,1,1,0,1). To save time, (3,2,1,1,0,0) is preferred because this layout provides the URMGC with the shortest return path. Therefore, in practice, the other scenarios within the homogeneous layout (3,2,1,1,0,0), such as (2,1,1,3,0,0) and (1,2,3,1,0,0), can be ignored. The layout with decreasing container counts from bay 1 to bay J is sufficient.

[0128] Similarly, taking the case of 8 containers as an example, we can also find a representative layout among multiple groups of homogeneous layouts, and summarize and simplify the original 1251 layouts into 18 groups, see Table 2.

[0129] Table 2: Simplified layout of 8 boxes in a bay

[0130]

[0131] This method identified the commonalities among different layouts within a bay, grouping 116,632 layouts into 812 homogeneous groups. A representative layout was selected for each group. This method effectively limited the number of constraints in the mathematical model to fewer than 1,000, significantly reducing the solution difficulty (see Table 3).

[0132] Table 3: Layout statistics after bay reduction

[0133]

[0134] Step S140, the upper bound of the case change after solving the robust peer-to-peer conversion by the deep reinforcement learning method.

[0135] For the equivalent deterministic 0-1 integer programming model, the upper bound of the case change corresponding to each layout τ needs to be obtained In the same layout, the future retrieval sequence of uncertainty will produce different case change times, and the upper bound of the case change The retrieval sequence corresponding to it is called the "worst retrieval sequence".

[0136] For the case of small number of containers n, the upper bound of the case change can be obtained by enumeration method, because the retrieval sequence can be enumerated, and the required case change times can be calculated. For example, the maximum case change number is 0 when the number is 1, and the maximum case change number is 1 when the number is 2, and so on. However, when the number of containers increases, for example, n = 10, there are kinds of retrieval sequences, and enumerating each retrieval sequence will consume a lot of computing resources, and the difficulty of solving will increase exponentially with the number of containers. In addition, when the number of containers in the bay is large, the problem of secondary case change needs to be considered, that is, after the initial case change operation, some containers may be moved again in the subsequent operation, which leads to the need for a certain amount of time to solve the case change number under a single retrieval sequence. Therefore, in an embodiment, an improved Actor-Critic deep reinforcement learning method is proposed to solve the upper bound of the case change of each layout under the same number of containers.

[0137] For example, the improved Actor-Critic deep reinforcement learning method includes two calculation modules: an accurate case change value calculation module and a compact upper bound value calculation module. The accurate case change value calculation module can be based on the 0-1 integer programming model proposed by Caserta et al. in 2012, and the accurate case change value calculation module can be solved by a commercial solver to provide the required accurate case change number under a specific layout and retrieval sequence. Because the mathematical model of this module is relatively complex, the solving time is relatively long.

[0138] In contrast, the compact upper bound value calculation module can give the same or close upper bound value as the accurate solution in a very short time, and the pseudo code thereof is shown in Table 4.

[0139] Table 4: Pseudo code of compact upper bound value calculation module

[0140]

[0141] Specifically, the compact upper bound value calculation method includes the following steps:

[0142] Step S11, initializing the box turning times counter n_relos to 0, for recording the total number of box turning times required in the operation.

[0143] Step S12: quality initialization.

[0144] For example, for each layer of each stack, the quality qlt[s][t] is set in the following way to reflect the priority order of the current stack: the quality of the first layer is set to the retrieval order pri[s][0] of the first layer of the current stack; for each subsequent layer t, the quality qlt[s][t] is set to the smaller value of the retrieval order pri[s][t] of the current layer and the quality qlt[s][t-1] of the previous layer. The quality value is used to guide how to choose the target stack for the blocked box when a box is pushed.

[0145] Step S13, when there are boxes in the stack that have not been processed, continue to perform the following steps: find the min_value of the box at the front of the retrieval order in all current stacks, and use it to determine the target box position (s_target, t_target) to be processed.

[0146] Step S14, determining whether the box is on the top floor.

[0147] If the target box is at the top of the stack, remove the box directly and update the corresponding retrieval order, quality, stack height and number of remaining boxes of the stack.

[0148] In step S15, if the target box is not at the top layer, a suitable target stack is selected to place the top blocked box in the original stack according to the following conditions: the target stack is not the current s_target to avoid circular placement; the height of the target stack is less than the maximum number of layers n_tiers; the quality of the top box of the target stack is greater than or equal to the search order of the current box to avoid the blocked box from being flipped twice in the target stack; if no stack fully meets the conditions, the qualified stack with the highest quality is selected as the second best choice.

[0149] Step S16, remove the blocked boxes and update the search order, quality, stack height and number of remaining boxes of the original stack.

[0150] In step S17, the blocked bin is placed on the target stack and the target stack's height, search order, and quality are updated. The quality is updated to the smaller of the current bin's search order and the lowest quality value of any other bin in the stack. This update ensures that the top-level quality reflects the highest-order bin in the stack, thus reducing the need to flip bins in the future.

[0151] Step S18: increase the box turning count n_relos.

[0152] Step S19, when all the bins in the stack are processed, return the total number of relocations n_relos.

[0153] Figure 7 An Actor-Critic deep reinforcement learning-based process is demonstrated for solving the relocation upper bound problem of representative layouts with the same number of containers. This method allows multiple layouts to be processed in parallel during a single training, and because the encoding length of each layout and its retrieval order is consistent, the data of different layouts has a unified representation in the network. This consistency not only reduces the complexity of encoding conversion, but also ensures that the network can treat the features of each layout as a vector of the same dimension during training, facilitating the sharing of weights in parallel processing. In addition, the consistent encoding length allows the network to efficiently utilize the commonalities between different layouts, thus converging more quickly to the global optimal strategy and eliminating the adaptation and alignment problems caused by encoding differences, further improving the model's generalization ability and training efficiency. Figure 7 The detailed description of each main module in Table 5.

[0154] Table 5: Description of each main module of Actor-Critic

[0155]

[0156] In combination Figure 7 As shown, the reinforcement learning-based relocation upper bound solving process includes the following steps:

[0157] Step S21, initial state setting: the current state of each layout is composed of the arrangement order of the bins and the layout identifier, which provides clear input information for the Actor and Critic networks.

[0158] Step S22, action selection: the Actor network outputs the probability distribution of actions based on the current state, and the selected action may involve exchange, inversion or insertion operation, aiming to optimize the retrieval order of the bins.

[0159] Step S23, state transition: the environment module receives the action generated by the Actor network, updates the current state to the new state (Next State) through the state transition module, and completes the adjustment of the retrieval order.

[0160] Step S24, immediate reward calculation: after the state transition, the compact upper bound calculation module calculates the difference in the number of relocations between the new and old states as the reward (Reward). This reward value reflects the immediate effect of the current action and provides feedback for policy adjustment. The compact upper bound calculation module uses the layout identifier to quickly match the pre-stored layout information, ensuring independent calculation of the reward value and providing a reliable basis for subsequent calculation of the TD Error.

[0161] Step S25, V value calculation: The Critic network estimates the value of the new state and the current state respectively, obtaining V(Next State) and V(Current State), which provides the basis for the calculation of the subsequent TD Error.

[0162] Step S26, TD Error calculation: TD Error is calculated as the feedback signal, and the formula is:

[0163] TD Error = Reward + γ * V(Next State) - V(Current State)

[0164] Step S27, Critic network parameter update: The Critic network calculates the loss (Loss) according to the TD Error, and adjusts the parameters through back propagation, and the update formula is:

[0165]

[0166] Through this update, the Critic network gradually optimizes the evaluation accuracy of each layout strategy, thereby providing more accurate feedback signals for the Actor network.

[0167] Step S28, Actor network parameter update: The Actor network optimizes its parameters according to the TD Error of the Critic network. If TD Error > 0, it means that the current action is better than expected, and the Actor network will increase the selection probability of the action; if TDError < 0, it means that the effect is lower than expected, and the Actor network will reduce the selection probability of the action. Through gradual adjustment, the Actor network continuously optimizes the strategy in the multi-layout environment, improving the overall convergence efficiency.

[0168] Step S29, termination condition judgment: After each iteration, check whether the preset termination condition (such as TDError less than or equal to the set threshold) is met. When the strategies of all layouts meet the termination condition, the training is completed, and the final retrieval strategy is input into the precise box value calculation module to ensure the accuracy of the model solution and the reliability of the actual application.

[0169] Step S150, optimize and solve the mathematical model through a commercial solver to obtain the container position allocation scheme and the terminal limit operation capacity under the current batch.

[0170] In order to further understand the present application, Figure 8The overall framework of the solution to the new automated terminal slot allocation strategy provided by the present invention is presented and explained as follows: a slot allocation model with uncertainty is constructed, which deeply considers the operating characteristics of the single cantilever rail crane (URMGC) and the automatic guided vehicle (AGV) and the changes in the new yard layout to ensure that the slot allocation decision is highly consistent with the actual operating environment. At the same time, it includes uncertain factors such as the order of picking up imported containers to more realistically reflect the challenges in actual operations; robust optimization is used to transform the uncertainty problem into a deterministic problem, and the original model is transformed into a deterministic problem by identifying and calculating the upper bound of the worst retrieval order. The model is converted into a robustly equivalent deterministic 0-1 integer programming model, enabling optimized slot allocation under the most unfavorable conditions. All possible layouts within a bay are statistically analyzed and classified to identify homogeneous layout groups, thereby reducing the model's complexity and the number of constraints, improving solution efficiency. An improved Actor-Critic deep reinforcement learning network is used to solve the specific value of the upper bound of container turnover, completing the data required for the mathematical model after reducing constraints. The model is solved using a commercial solver (or precise algorithm) to obtain the terminal's ultimate operating capacity and optimal slot allocation solution for the current batch. This series of steps, working together, provides a systematic process for solving the uncertain slot allocation problem for import containers, ensuring the practical applicability and efficiency of the solution and providing reliable technical support for terminal operators to efficiently manage terminal operations.

[0171] In summary, this paper proposes an innovative slot allocation method that combines deep reinforcement learning with robust optimization to transform an uncertain model into a deterministic one, significantly improving solution efficiency and real-time performance. Furthermore, by solving the import container allocation scheme under the most unfavorable future search order, terminal operators can not only rationally optimize slot allocation but also effectively assess the terminal's container area operating capacity limits when handling current import containers, providing strong decision support for terminal operations management.

[0172] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0173] Computer-readable storage medium can be a tangible device that can keep and store the instructions used by the instruction execution device.Computer-readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device or any suitable combination thereof.More specific examples (non-exhaustive list) of computer-readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove having instructions stored thereon, and any suitable combination thereof.Computer-readable storage medium used herein is not interpreted as a transient signal itself, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagated by waveguides or other transmission media (for example, light pulses by fiber optic cables), or electrical signals transmitted by wires.

[0174] Various aspects of the present invention are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present invention. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0175] The flowcharts and block diagrams in the accompanying drawings show the possible implementation architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flowchart or block diagram can represent a module, program segment or part of an instruction, and the module, program segment or part of the instruction contains one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that implementation by hardware, implementation by software, and implementation by a combination of software and hardware are all equivalent.

[0176] While various embodiments of the present invention have been described above, the foregoing description is intended to be illustrative, non-exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is selected to best explain the principles of the embodiments, their practical applications, or technological improvements in the marketplace, or to enable others skilled in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for allocating import container slots at an automated terminal embedded with deep reinforcement learning, comprising the following steps: For the container pickup area, an objective function is established based on the total processing time of the current batch of containers and the total time required to pick up all containers in the future. Performing robust equivalent transformation on the objective function to obtain a deterministic mathematical model; Using deep reinforcement learning methods to solve the upper bound of the objective function in the deterministic mathematical model; The slot allocation solution is obtained by optimizing the solution with a commercial solver. Wherein, the objective function is set as: The constraints for solving the objective function are set as: in: is the import box index, , is the number of imported boxes; is the stack index, , is the number of stack positions; is the bay index, , is the number of bays; It is the time it takes for the AGV to travel to the yard after loading a container from the shore crane; It is the time it takes for the AGV to return from the yard to the quay crane to wait for the container handover; is the unit movement time of the AGV with the container; is the unit moving time of the unloaded AGV; It is the unit horizontal movement time of a single-sided cantilever rail gantry crane between stacking locations; It is the unit horizontal movement time of the single-side cantilever rail gantry crane between bays; It is the floor height; is the capacity in beta; It is a stack position , Bay The initial number of containers in ; is the unit time of the box turning operation; is a decision variable, if the import box Assigned to the stack Bay position ,but ,otherwise, ; It is an imported box Assigned to the stack; It is an imported box Assigned to a bay; It is an imported box The total handling time, representing the time from the container Waiting for the automated guided vehicle to pick up the container at the quay crane until the automated guided vehicle hands over the container to the single-side cantilever rail gantry crane; After the slots are allocated, the current slot The layout corresponds to the upper bound of the number of box turns required when picking up boxes in the future; and is the weighting coefficient of the corresponding term.

2. The method according to claim 1, characterized in that The objective function is simplified as: Compared with the objective function, the following constraints are added: in: is the layout index contained in the bay, , is the sum of all bay layouts; Different bay layouts The symbolic value of Different bay layouts The upper bound of the flip box; is a decision variable, if The layout in ,but ,otherwise, .

3. The method according to claim 2, characterized in that The upper limit of the box turning Obtained through deep reinforcement learning, it includes the following steps: We use an actor-critic deep reinforcement learning approach to calculate the search order required to find the upper bound of the exact bin flipping problem. Based on the calculated retrieval order, a 0-1 integer programming model is used to obtain the exact solution of the retrieval order under the set layout using a commercial solver as the upper bound of the box flipping.

4. The method according to claim 3, characterized in that The actor-critic deep reinforcement learning method is used to calculate the search order required to solve the upper bound of the exact box flipping problem, including the following steps: Set the initial state, where the current state of each bay layout is composed of the container arrangement order and layout identifier; The actor network outputs a probability distribution of actions based on the current state. The selected actions include swap, reversal, or insertion operations. The environment module receives the action generated by the actor network and updates the current state to the new state Next State through the state transition module; The compact upper bound calculation module calculates the compact upper bound difference between the new and old states regarding the box flipping as a reward value Reward, which reflects the immediate effect of the current action, and the compact upper bound calculation module uses the layout identifier to match the pre-stored layout information; The critic network estimates the value of the new state and the current state respectively, and obtains the value estimate of the new state and the value estimate of the current state; Calculate the time difference error reflecting the difference between the current state and the expected state as the feedback signal, and backpropagate the feedback signal to the actor network and the critic network to adjust the parameters within the actor network and the critic network; When the time difference error meets the preset termination condition, the retrieval order learned at this time is output.

5. The method according to claim 4, wherein The time difference error is calculated according to the following formula: TD Error = Reward+ *V(Next State) -V(Current State) Among them, TD Error represents the time difference error, Reward represents the reward value, V (Next State) is the value estimate of the new state, and V (Current State) is the value estimate of the current state. is the weighting coefficient of the value estimate of the new state.

6. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.

7. A computer device comprising a memory and a processor, wherein a computer program capable of being run on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.

Images