Automatic wharf import container position distribution method with embedded deep reinforcement learning

Abstract

The invention discloses an automatic wharf import container position distribution method with embedded deep reinforcement learning. The method comprises the steps that firstly, for a container lifting area, an objective function is established based on the total processing time of a current batch of containers and the total turnover time of all containers in the future; then, robust peer-to-peer conversion is carried out on the established mathematical model, an uncertainty model is converted into a deterministic model, and constraints of position layout selection are correspondingly added; thirdly, carrying out statistics and simplification on the possible layout of the bay positions so as to greatly reduce the number of constraint conditions in the model; and solving a container turnover upper bound value in the objective function after robust peer-to-peer conversion through a deep reinforcement learning method, and finally obtaining a container position distribution scheme by utilizing a commercial solver. According to the method, on the basis of remarkably improving the solving efficiency of the container position distribution strategy, it is ensured that the wharf has higher stability and reliability when facing the import container operation with high uncertainty, and powerful decision support is provided for wharf operation management.

Description

Technical Field

[0001] The present invention relates to the technical field of container scheduling, and more specifically, to an automated terminal import container slot allocation method embedded with deep reinforcement learning. Background Art

[0002] As the global supply chain continues to develop, major ports have upgraded their container terminals to improve port efficiency. The rise of new automated container terminals marks a new stage in port automation. A notable feature of these new automated terminals is the change in yard crane equipment. The new automated terminal yard crane mainly uses a single-sided cantilever rail gantry crane (URMGC). Compared with the automatic stacking crane (ASC) widely used in traditional automated terminals, the cantilever part of the URMGC extends to one side of the crane, increasing the lateral movement range of the crane. The introduction of this equipment has brought about significant changes in the yard layout. For example Figure 1 As shown in the figure, in the yard layout of the new automated terminal, the passages between adjacent container areas are open to the automated guided vehicle (AGV), that is, the AGV can drive into the yard area and interact with the crane under the protruding end of the cantilever. In the traditional automated terminal, the AGV can only interact with the crane at the end of the yard (on the water side). The significant change in the yard layout has triggered a fundamental change in the yard operation mode. Figure 1 For example, in the new automated terminal, containers entering the yard from the water side mainly rely on automatic guided vehicles, which can significantly reduce the travel of the yard crane. In traditional automated terminals, containers entering the yard can only rely on the longitudinal movement of the yard crane along the guide rail from the water side to the land side. Since the moving speed of the gantry yard crane is much lower than that of the AGV, this means that the new automated terminal has a higher single-box handling efficiency.

[0003] Import container slot allocation is an important decision that port enterprises need to make in terminal operations. Reasonable slot allocation can effectively help port enterprises reduce costs and improve terminal efficiency, and is of great significance to alleviate key issues that hinder the convenience of global trade, such as long waiting time for ships and port congestion. Specifically, the import container slot allocation problem refers to the allocation of an available slot for each container arriving at the port within a certain period of time, based on the equipment characteristics, layout characteristics, existing slot resources of the current yard, and considering uncertain factors such as the time and order of container pickup in the future, so as to minimize the total terminal operation time (time cost) for a given number of import containers. Terminal operations usually include two stages. The first stage is the stacking operation, that is, stacking a given number of import containers to the designated slot. The second stage is the container pickup operation, that is, the terminal extracts the designated container from the yard so that the cargo owner can pick up the goods.

[0004] The existing technology is mainly aimed at traditional automated terminal scenarios, with the goal of minimizing terminal operation time (or time cost). First, a mathematical programming model that takes uncertainty into account is constructed for the allocation of imported container slots, and then a solution method is designed for the model. In terms of mathematical models, the existing technology's characterization of the stacking stage is based on the characteristics of the interaction between the AGV and the yard crane on the water side of the traditional automated terminal; the characterization of the container picking stage mainly considers the additional container turning operations caused by the uncertainty of the container picking information. In terms of solution methods, the existing technology mainly solves the model based on random programming methods. For example, the probability distribution of the relevant parameters of the container picking information is first fitted according to historical data, and then multiple container picking scenarios are generated based on the probability distribution. The container slot allocation problem considering uncertainty is transformed into a deterministic model with the goal of minimizing the expectation of terminal operation time under multiple scenarios, and finally solved using commercial solvers or precise algorithms.

[0005] After analysis, the existing technology is not suitable for the new automated terminal. This is because the biggest feature of the new automated terminal in the operation of imported containers is that the AGV and the yard bridge can interact in the yard channel, and the mathematical model of the existing technology is based on the traditional automated terminal scenario and cannot capture the interactive characteristics of the new terminal. In addition, when solving the container allocation model considering uncertainty, the method based on random optimization mainly adopts the expectation planning based on multiple scenarios to deal with the uncertainty of the container picking stage, but the container allocation decision provided by this type of method has not produced the expected effect in reducing the additional container turning operations in the container picking stage in practice. This is because: First, due to the high uncertainty of the container picking information (for example, the time and order of container picking), the distribution fitting of the relevant parameters cannot accurately reflect the statistical characteristics of the overall sample of the container picking information. This means that there are a large number of scenarios in actual operation that cannot be simulated by existing technologies. Second, taking the expectation of terminal operation time under multiple scenarios as the optimization target is essentially based on the probability weighted scenario to make container allocation decisions. This will cause the number of container turning operations to be too sensitive to changes in container picking information. Specifically, if the actual box-picking information meets the box-picking scenario with a greater probability weight, that is, a box-picking scenario with a higher frequency, the box-slot allocation scheme provided by the prior art can provide better feedback (less box-picking times when picking up boxes). If the actual box-picking information meets the box-picking scenario with a smaller probability weight, that is, a box-picking scenario with a lower frequency, the box-picking allocation scheme provided by the prior art may not be able to reduce or even increase the number of box-picking times. Summary of the invention

[0006] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and provide an automated terminal import container slot allocation method embedded with deep reinforcement learning. The method comprises: for the container pick-up area, an objective function is established based on the total processing time of the current batch of containers and the total time required to turn over all containers in the future; the objective function is robustly converted to obtain a simplified objective function with determinism; the simplified objective function is solved by the reinforcement learning method to obtain a slot allocation plan.

[0007] Compared with the prior art, the advantage of the present invention is that it constructs a new slot allocation model, deeply considers the operating characteristics of the single cantilever rail crane (URMGC) and the automatic guided vehicle (AGV), as well as the changes in the new yard layout, ensuring that the slot allocation decision is highly consistent with the actual operating environment, and significantly improving the terminal's operating efficiency and resource utilization. In addition, the method proposed in the present invention is based on robust optimization, and the slot allocation plan is determined by decision, which fully considers the extremely complex container turning conditions that may be caused by the flexible and changeable order of picking up the containers, and provides time (time cost) redundancy for actual terminal operations, ensuring that the terminal efficiency under the decision-making plan has higher stability and reliability in the face of highly uncertain import container operations.

[0008] Further features and advantages of the present invention will become apparent from the following detailed description of exemplary embodiments of the present invention with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0010] Figure 1 is a schematic diagram of different automated container terminal layouts of the prior art;

[0011] Figure 2 is a flowchart of an automated terminal import container slot allocation method embedded with deep reinforcement learning according to one embodiment of the present invention;

[0012] Figure 3 is a schematic diagram of a container yard area according to an embodiment of the present invention;

[0013] Figure 4 is a schematic diagram of the plan layout of a new automated container terminal according to an embodiment of the present invention;

[0014] Figure 5 is a schematic diagram of a box turning operation according to an embodiment of the present invention;

[0015] Figure 6 is a three-dimensional schematic diagram of a novel automated terminal yard operation according to an embodiment of the present invention;

[0016] Figure 7 is a flowchart of solving the upper bound of a box flipping according to an embodiment of the present invention;

[0017] Figure 8 It is a schematic diagram of a novel automated terminal slot allocation solution framework according to an embodiment of the present invention. DETAILED DESCRIPTION

[0018] Various exemplary embodiments of the present invention will now be described in detail with reference to the accompanying drawings. It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values ​​set forth in these embodiments do not limit the scope of the present invention unless otherwise specifically stated.

[0019] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.

[0020] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered as part of the specification.

[0021] In all examples shown and discussed herein, any specific values ​​should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0022] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0023] This paper proposes an innovative method for allocating slots for import containers. The method mainly includes: a slot allocation model that adapts to the layout of new automated terminals and the characteristics of equipment interaction; a model solving method based on the idea of ​​robust optimization and integrating deep reinforcement learning technology. The slot allocation method aims to solve the following problems:

[0024] 1) Considering the defects of the existing slot allocation model that cannot adapt to the new automated terminal, a new import container slot allocation model considering uncertainty is constructed. The model aims to minimize the operation time of the new automated terminal, fully considers the yard layout of the new automated terminal, the cantilever operation of the URMGC and the movement characteristics of the AGV in the yard, can accurately describe the yard layout of the new automated terminal, and the characteristics of the interaction between the yard crane and the AGV in the yard, ensuring that the model can accurately calculate and optimize the operation time of the container stacking stage of the new terminal.

[0025] 2) A model solving method based on the idea of ​​robust optimization and integrated with deep reinforcement learning technology is designed. Due to the container allocation scheme determined by the existing technology, there is a problem that the number of container turning operations in the container picking operation is too sensitive to changes in the container picking information. Based on the idea of ​​robust optimization, the present invention designs a solution method for the container allocation model that takes into account uncertainty, and considers the maximum limit of the number of container turnings that may be caused by the uncertainty of the container picking order. It ensures that the terminal operation time under this extreme situation can still be optimized, thereby reducing the high sensitivity of the container turning operation to changes in the container picking order.

[0026] In general, the automated terminal import container slot allocation method embedded with deep reinforcement learning provided by the present invention includes the following aspects:

[0027] 1) For a given container stacking layout, find the order of picking up containers (i.e., the retrieval order) with the most container turnovers under the stacking layout. Define this retrieval order as the worst retrieval order, and the corresponding number of container turnovers is the upper bound of the number of container turnovers under the stacking layout.

[0028] 2) Using the mapping relationship between stacking layout and the upper limit of the number of box turning, constraints are established for all possible stacking layouts. Since there are many possible stacking layouts, if the stacking layout is adopted, the number of box turning must not exceed its corresponding upper limit. Since this constraint ensures that no matter how the box picking order changes, it will not exceed the maximum number of box turning, eliminating the uncertainty of the box picking order, so the constraint can be used to transform the model considering uncertainty into a deterministic model.

[0029] 3) Solve the deterministic model using an exact or heuristic optimization algorithm, and finally obtain the bin allocation solution. Since the combination scale of "stacking layout-bin picking order" is very large, it is almost impossible to determine the worst retrieval order and the upper limit of the number of bin flipping by traversing all combinations, which will lead to dimensionality disaster. Therefore, deep reinforcement learning technology is introduced. Based on finite-dimensional data, a model that can capture the mapping relationship of "stacking layout-worst retrieval order" is trained, and the model is used to determine the upper limit of the number of bin flipping for a given stacking layout.

[0030] For example, based on the Actor-Critic framework, the Actor neural network is used to generate the action or action probability distribution (such as exchange, reversal, insertion, etc.) in the current state to optimize the retrieval order strategy; at the same time, the Critic neural network is used to evaluate the value (V value) of the current state and action pair. The Actor network and the Critic network are trained collaboratively through the calculated TD Error (time difference error), and the parameters are gradually adjusted until the TD Error converges, thereby achieving strategy optimization and approximation of the global optimal solution.

[0031] For another example, a custom compact upper bound evaluation module is introduced to accelerate the training of Actor-Critic. This module can provide timely reward evaluation for Actor-Critic by quickly calculating the difference in the compact upper bound number of container flipping between the current state and the new state. In addition, in multi-layout collaborative training for the same number of containers, this module quickly matches the pre-stored layout information through the layout identifier and independently generates reward values ​​for specific layouts. This method significantly speeds up the training convergence process and shortens the overall time required to implement the strategy. When the training reaches the termination condition, the final retrieval order will be input into the precise container flipping value calculation module to ensure the accuracy of the container allocation model during the solution process and to ensure that the resulting solution is highly reliable in practical applications.

[0032] Specifically, see Figure 2 As shown, the provided automated terminal import container slot allocation method embedded with deep reinforcement learning includes the following steps:

[0033] Step S110, constructing a mathematical model for slot allocation at a new automated container terminal, and establishing an objective function based on the total processing time of the current batch of containers and the total container turnover time required to take away all containers in the future.

[0034] 1. Problem description

[0035] In the new automated container terminal, the yard provides a temporary storage place for the large number of containers that need to be circulated in this terminal. The yard is composed of a block, and the containers are stored in a specific position in the block, such as Figure 3 shown.

[0036] In traditional automated container terminals, automated guided vehicles (AGVs) are only responsible for transporting containers to the docking areas on both sides of the container area, and there are no roads inside the container area for AGVs to move. The vertical movement of containers assigned to specific bays and the horizontal movement between stacks are all completed by cranes. In new automated terminals, AGVs can already penetrate deep into the container area to achieve precise control and positioning. When AGVs perform container storage tasks, they are responsible for moving containers to specific bays, while the movement of containers between stacks is undertaken by cranes. Using faster-moving AGVs to replace cranes in moving between bays can effectively improve the overall operating efficiency of the terminal, see Figure 4 The floor plan of the new automated container terminal is shown.

[0037] After the imported container is unloaded from the ship by the quay crane, the internal container truck (AGV) transports it from the coast to the designated bay. Subsequently, the yard crane unloads the container from the internal container truck and temporarily stores it in the specific stack corresponding to the bay in the container area, waiting for the external container truck to pick it up. When picking up the container in the future, the yard crane will hand over the target container from the storage location to the external container truck for transportation to the inland hinterland.

[0038] When the yard crane unloads the imported container from the internal container truck and temporarily stores it in a certain container space in the container area, the yard operator needs to select the most reasonable storage location for the current container among all the container spaces in the container area that have not yet stored containers. This decision-making problem of selecting the most appropriate storage location for the imported container is called the imported container space allocation problem.

[0039] Whether the slot allocation plan for a large number of imported containers is reasonable not only directly affects the time that the liner waits to unload the containers, but also indirectly affects the time that the foreign container truck will stay in the container area handover area in the future. This is because when picking up containers in the future, the carrier responsible for the task of picking up containers is a highly flexible foreign container truck, rather than a highly planned ship. Since the terminal cannot determine the arrival order of foreign container trucks, it is impossible to predict the order of picking up imported containers in advance. This may cause the target container to be pressed down by other containers when picking up the container. After the foreign container truck arrives at the handover area, it needs to wait for the yard crane to turn the container over before it can pick up the target container.

[0040] Container flipping means that when the target container is covered by other containers (blocking containers), the yard crane must first move the blocking container above to a stack with vacant container space before successfully extracting the target container. Figure 5 As shown in the figure, it is assumed that the target container for the current external container truck to pick up is box C1 located in stack 3, and box C1 is located below boxes C2, C3, and C4. The yard crane has to first move box C4 at the top of stack 3 to stack 2 (the first container turning operation), then move box C3 to stack 4 (the second container turning operation), and finally move box C2 to stack 4 (the third container turning operation), before box C can be successfully taken out and handed over to the external container truck. Obviously, the container turning operation will increase the total time consumption of the container picking operation, prolong the waiting time of the container truck in the handover area, and reduce the work efficiency and service level of the terminal.

[0041] For terminal operators, when container pick-up information is highly uncertain, they need to reasonably consider the allocation of import container slots, which is the key to improving the container turnover efficiency of the terminal, solving the bottleneck of operational efficiency, and improving customer service levels and satisfaction.

[0042] 2. Mathematical model for import container slot allocation

[0043] Basic Assumptions

[0044] 1) Containers of the same batch will be assigned to the same container area.

[0045] 2) The number of AGVs assigned to a container area is limited, which is set to one in this model.

[0046] 3) Container storage and container pickup operations will not be carried out at the same time. Container pickup operations will only begin after the container slot allocation is completed.

[0047] 4) Container types are homogeneous.

[0048] 5) The order in which each imported container arrives at the port is known in advance.

[0049] index

[0050] i: Import container index, i∈{1,2,…,I}. Import containers are numbered in the order of arrival.

[0051] j: stack position index, j∈{1,2,…,J}. Stack position 1 is on the left side of the bin, and stack position j is on the right side of the bin.

[0052] k: Bay index, k∈{1,2,…,K}. Bay 1 is close to the land side, and bay k is close to the water side.

[0053] parameter

[0054] e1: The time it takes for the AGV to travel to the yard after loading the container from the shore crane.

[0055] e2: The time it takes for the AGV to return from the yard to the quay crane and wait for the container to be delivered.

[0056] n1: Unit moving time of the AGV equipped with the container.

[0057] n2: Unit moving time of the unloaded AGV.

[0058] m1: URMGC unit horizontal movement time between stack locations.

[0059] m2: Unit horizontal movement time of URMGC between bays.

[0060] H: Floor height.

[0061] C: capacity in bainite.

[0062] θ jk : The initial number of containers in stack position j, stack position k. j∈{1,2,…,J}, k∈{1,2,…,K}.

[0063] r: Unit time of box turning operation.

[0064] Decision variables

[0065] x ijk : If import box i is allocated to stack position j, shell position k, then x ijk = 1. Otherwise, x ijk = 0. Where i∈{1,2,…,I}, j∈{1,2,…,J}, k∈{1,2,…,K}.

[0066] Dependent Variable

[0067] Si : Import box i is assigned to a stack position, i∈{1,2,…,I}.

[0068] L i : Import bin i is assigned to a bin position, i∈{1,2,…,I}.

[0069] T i : The total processing time of imported container i, that is, from the time when container i waits for AGV to pick up the container at the quay crane to the time when AGV hands over the container to URMGC. i∈{1,2,…,I}.

[0070] R(θ jk ;x ijk ):After the slots are allocated, the layout of the current slot k corresponds to the upper bound of the number of times the slots will be turned over when picking up the containers in the future.

[0071] Objective Function

[0072] In one embodiment, the objective function is set to:

[0073]

[0074] The objective function describes the operational capacity limit of the container area when handling the current liner, which consists of two objectives: one is the total processing time of the current batch of containers, and the other is the total time required to turn over all containers in the future.

[0075] is about the decision variable x ijk In practice, for the same number of containers in a bay, if the layout is different, the number of container flipping will be very different. For example, for a bay with six layers and six stacks, if there are six containers in the bay, if the six containers are laid flat, no container flipping will occur. However, if all six containers are placed in the same stack, then container flipping will occur except for the case where the retrieval order is arranged from bottom to top.

[0076] Therefore, in view of the fundamental difference in the number of box turning caused by this layout, the present invention improves the box turning function. That is, one layout corresponds to one number of box turning.

[0077] Constraints

[0078]

[0079] T1=(K-L1)n1+e1 (5)

[0080]

[0081]

[0082] Constraint (2) means that any container in the same batch must be allocated to a container area;

[0083] Constraints (3) and (4) are used to calculate the stack and bay positions to which each container is assigned;

[0084] Constraint (5) indicates that for the first container, after determining its assigned location, the field crane will wait at the bay in advance. At this time, the time it takes for the AGV to enter the aisle after loading the container on the water side and drive to a specific bay to hand it over to the field crane is the operation time of the first container;

[0085] Constraint (6) indicates that for subsequent containers, after the specific location to be allocated is determined, its operation time will be the maximum between the operation time of the field crane and the AGV. The operation time of the field crane is the time from the last container to the location where the current container will be allocated. The operation time of the AGV is the time from the location where the last container was transferred to the water side to load the current container and then drive to the designated location.

[0086] Constraint (7) indicates that the stacking of containers cannot exceed the layer height;

[0087] Constraint (8) indicates that the number of containers in the same bay cannot be too large. This is because when taking containers in the future, there may be a container flipping operation, so a certain amount of space needs to be reserved.

[0088] Constraint (9) defines a 0-1 decision variable regarding the allocation of slots.

[0089] Step S120, performing robust equivalent transformation on the established objective function to obtain a deterministic mathematical model, and correspondingly adding constraints on the selection of bay layout.

[0090] In the mathematical model, the box flipping function Represented as a layout, corresponding to a number of box turning.

[0091] However, for imported containers, since the container retrieval task is completed by flexible external trucks, the terminal cannot determine the arrival order of these trucks, and therefore cannot predict the order of retrieval of imported containers. Although the layout is determined, the uncertainty of the retrieval order makes It is still impossible to directly calculate the corresponding number of box flipping times.

[0092] For this uncertainty model, in one embodiment, a robust equivalence conversion is used to determine the worst search order and calculate the corresponding number of container turnovers for each specific layout in the bay, and the model is transformed into a solvable deterministic model. That is, on the basis of considering the most unfavorable situation, the operational capacity limit of the container area when handling the current liner containers is further evaluated. The process of robust equivalence conversion is as follows:

[0093] The objective function of the original model [OP] can be simplified as:

[0094]

[0095] By using a deterministic approach, we consider the worst retrieval order under a fixed layout within the bay:

[0096] R(·)~layout→U R (layout)

[0097] Among them, U R (·) represents the upper bound of the box flipping, that is, the box flipping value corresponding to the worst search order. For the n layouts contained in the bay, their upper bounds can be found:

[0098] R(l1)≤U R (l1)

[0099] R(l2)≤U R (l2)

[0100] …

[0101] R(l n )≤U R (l n )

[0102] For the box flipping function R(·) with uncertainty, according to the definition of robust optimization:

[0103]

[0104] Among them, l is the decision variable, that is, the decision variable x in the box allocation decision ijk and parameter θ jk The layout composed of is defined in the feasible domain L; s is an uncertainty variable, that is, the search order, which is defined in the set S of all possible search orders. s∈S R(l,s), takes the maximum value, that is, the upper bound of the bin flipping, removing the uncertainty caused by the retrieval order:

[0105]

[0106] For the original mathematical model, new parameters and decision variables are introduced to characterize the process.

[0107] Add index

[0108] τ: The possible layout index in the bay, τ∈{1,2,…,σ}. σ is the sum of all possible layouts.

[0109] New parameters

[0110] ω τ : Symbolic values ​​of different layouts τ, τ∈{1,2,…,σ}.

[0111] Upper bounds of box flipping for different layouts τ, τ∈{1,2,…,σ}.

[0112] Adding decision variables

[0113] y τk : If the layout in bay k is τ, then y τk =1. Otherwise, y τk = 0. Where k∈{1,2,…,K}, τ∈{1,2,…,σ}.

[0114] The robust equivalence problem of the original model is:

[0115]

[0116] Add constraints

[0117]

[0118] The objective function (10) takes into account the optimization of the container allocation for a batch of imported containers under the most unfavorable retrieval conditions in the future for the current container area. The value to be calculated is the maximum operating capacity of the container area under the current batch.

[0119] Constraints (11) and (12) together indicate that among all possible layouts, the node can only choose one of them.

[0120] Constraints (13) define 0-1 decision variables regarding layout selection.

[0121] Step S130, statistics and simplification of possible layouts of bays are performed to significantly reduce the number of constraints in the mathematical model.

[0122] For the possible layout τ in a bay, it is necessary to count all possible layouts σ and reasonably represent them. In a bay, there are generally six stacks with a height of six layers. Taking this as an example, all layouts containing 1-25 containers in a bay are counted, see Table 1.

[0123] Table 1: Bay location statistics

[0124]

[0125] As can be seen from the table, for a bay with six stacking positions and layer heights, there are about 100,000 layouts when the bay capacity is 25, considering the box turning operation and safety factors. In the mathematical model, if the search order is not considered, this will introduce about 100,000 constraints, which will bring great difficulties to the construction and solution of the model. In one embodiment, it is proposed to solve this problem by summarizing and simplifying the layout.

[0126] Take the simplest case as an example: when there is only one container in the bay, there are six layouts: (1,0,0,0,0,0), (0,1,0,0,0,0), (0,0,1,0,0,0), (0,0,0,1,0,0), (0,0,0,1,0,0), (0,0,0,0,1,0) and (0,0,0,0,0,1). These layouts have the same elements (i.e., they all contain 5 0s and 1 1), but only have different order of arrangement, so they are called "homogeneous layouts". After these layouts are classified into homogeneous groups, they can share the same mapping relationship between the search order and the number of container turnovers.

[0127] On the other hand, Figure 6 As shown in the figure, in the container area of ​​the new automated terminal, assuming that the URMGC and AGV place the container at the designated bay (2,2,1,1,0,0), they need to choose from the layouts (3,2,1,1,0,0), (2,3,1,1,0,0), (2,2,2,1,0,0), (2,2,1,2,0,0), (2,2,1,1,1,0), (2,2,1,1,0,1). In order to save time, (3,2,1,1,0,0) will be preferred. This is because the return path of the URMGC is the shortest under this layout. Therefore, other situations such as (2,1,1,3,0,0), (1,2,3,1,0,0) in the homogeneous layout (3,2,1,1,0,0) can be ignored in actual operation, and the layout with the number of containers decreasing from bay 1 to bay J can be retained.

[0128] Similarly, taking the case where the number of containers is 8 as an example, we can also find representative layouts among multiple groups of homogeneous layouts, and summarize and simplify the original 1251 layouts into 18 groups, see Table 2.

[0129] Table 2: Simplified layout of 8 boxes in the bay

[0130]

[0131] Through this method, the commonalities of different layouts within the bay are identified, 116,632 layouts are summarized into 812 homogeneous layout groups, and a representative layout is selected for each group. This method effectively controls the constraints of the mathematical model to within a thousand, greatly reducing the difficulty of solving the problem, see Table 3.

[0132] Table 3: Layout statistics after bay reduction

[0133]

[0134] Step S140, after solving the robust equivalent transformation through the deep reinforcement learning method, the upper bound of the box flipping in the objective function is obtained.

[0135] For the equivalent deterministic 0-1 integer programming model, it is necessary to find the upper bound of the box flipping corresponding to each layout τ In the same layout, the uncertain future box retrieval order will produce different box flipping times, and the upper bound of the box flipping The corresponding search order is called the "worst search order".

[0136] For a small number of containers n, the upper bound of the container turnover It can be obtained by enumeration, because the search order It can be enumerated and the number of container turns required can be calculated. For example, when the number is 1, the maximum number of container turns is 0, when the number is 2, the maximum number of container turns is 1, and so on. However, when the number of containers increases, for example, n = 10, There are many search sequences. Enumerating each search sequence will consume a lot of computing resources, and the difficulty of solving will increase exponentially with the number of containers. In addition, when there are many containers in a bay, the secondary container turning problem needs to be considered, that is, some containers may be moved again in subsequent operations after the initial container turning operation, which results in a certain amount of time required even for solving the number of container turning under a single search sequence. Therefore, in one embodiment, an improved Actor-Critic deep reinforcement learning method is proposed to solve the upper bound of container turning for various layouts with the same number of containers.

[0137] For example, the improved Actor-Critic deep reinforcement learning method includes two calculation modules: the precise box flipping value calculation module and the compact upper bound value calculation module. The precise box flipping value calculation module can be solved by a commercial solver based on the 0-1 integer programming model proposed by Caserta et al. in 2012, and can provide the required precise box flipping times under a specific layout and retrieval order. Since the mathematical model of this module is relatively complex, the solution time is relatively long.

[0138] In contrast, the compact upper bound calculation module can give an upper bound value that is the same or close to the exact solution in a very short time. Its pseudo code is shown in Table 4.

[0139] Table 4: Pseudo code of the compact upper bound calculation module

[0140]

[0141] Specifically, the compact upper bound value calculation method includes the following steps:

[0142] Step S11, initializing the box turning times counter n_relos to 0, which is used to record the total number of box turning times required in the operation.

[0143] Step S12: quality initialization.

[0144] For example, for each layer of each stack, the quality qlt[s][t] is set in the following way to reflect the priority order of the current stack: the quality of the first layer is set to the retrieval order pri[s][0] of the first layer of the current stack; for each subsequent layer t, qlt[s][t] is set to the smaller value of the retrieval order pri[s][t] of the current layer and the quality qlt[s][t-1] of the previous layer. The quality value is used to guide how to choose the target stack for the blocked box when a box is pushed.

[0145] Step S13, when there are boxes in the stack that have not been processed, continue to perform the following steps: find the min_value of the box that is at the front of the search order in all current stacks, which is used to determine the position of the target box to be processed (s_target, t_target).

[0146] Step S14, determining whether the box is on the top floor.

[0147] If the target box is at the top of the stack, the box is removed directly and the corresponding retrieval order, quality, stack height and number of remaining boxes of the stack are updated.

[0148] Step S15, if the target box is not at the top layer, it is necessary to select a suitable target stack to place the top blocked box in the original stack according to the following conditions: the target stack is not the current s_target to avoid circular placement; the height of the target stack is less than the maximum number of layers n_tiers; the quality of the top box of the target stack is greater than or equal to the retrieval order of the current box to avoid secondary box flipping in the target stack; if there is no stack that fully meets the conditions, select the qualified stack with the highest quality as the second best choice.

[0149] Step S16, remove the blocked boxes and update the search order, quality, stack height and number of remaining boxes of the original stack.

[0150] Step S17, the blocked box is placed on the target stack, and the height, search order, and quality value of the target stack are updated. The quality is updated to the smaller value between the search order of the current box and the lowest quality value of other boxes in the stack. This update ensures that the top quality reflects the box with the highest search order in the stack, thereby reducing the need to turn boxes in the future.

[0151] Step S18, increasing the box turning times count n_relos.

[0152] Step S19, when all boxes in the stack have been processed, the total number of box turning times n_relos is returned.

[0153] Figure 7 The process based on Actor-Critic deep reinforcement learning is demonstrated to solve the problem of solving the upper bound of container turning for each representative layout under the same number of containers when the number is large. This method allows multiple layouts to be processed in parallel in one training, and because the encoding length of each layout and its retrieval order is consistent, the data of different layouts have a unified representation in the network. This consistency not only reduces the complexity of encoding conversion, but also ensures that the network can treat the features of each layout as vectors of the same dimension during training, which facilitates the sharing of weights in parallel processing. In addition, the consistent encoding length enables the network to efficiently utilize the commonalities between different layouts, thereby converging to the global optimal strategy faster, eliminating the adaptation and alignment problems caused by encoding differences, and further improving the generalization ability and training efficiency of the model. Figure 7 The detailed description of each major module is shown in Table 5.

[0154] Table 5: Description of the main modules of Actor-Critic

[0155]

[0156] Combination Figure 7 As shown in the figure, the process of solving the upper bound of box flipping based on reinforcement learning includes the following steps:

[0157] Step S21, initial state setting: The current state of each layout is composed of the arrangement order of the boxes and the layout identifier, providing clear input information for the Actor and Critic networks.

[0158] Step S22, action selection: The Actor network outputs the probability distribution of actions based on the current state. The selected action may involve swap, reversal, or insertion operations, aiming to optimize the retrieval order of boxes.

[0159] Step S23, state transition: the environment module receives the action generated by the Actor network, updates the current state to a new state (Next State) through the state transition module, and completes the adjustment of the retrieval order.

[0160] Step S24, immediate reward calculation: After the state transition, the compact upper bound calculation module calculates the upper bound difference of the number of box flips between the new and old states as a reward. The reward value reflects the immediate effect of the current action and provides feedback for strategy adjustment. The compact upper bound calculation module uses the layout identifier to quickly match the pre-stored layout information to ensure the independent calculation of the reward value, providing a reliable basis for the subsequent calculation of TD Error.

[0161] Step S25, V value calculation: The Critic network estimates the value of the new state and the current state respectively, and obtains V (Next State) and V (Current State), which provide a basis for the subsequent TD Error calculation.

[0162] Step S26, TD Error calculation: TD Error is used as a feedback signal, and the calculation formula is:

[0163] TD Error=Reward+γ*V(Next State)-V(Current State)

[0164] Step S27, Critic network parameter update: The Critic network calculates the loss (Loss) according to the TD Error and adjusts the parameters through back propagation. The update formula is:

[0165]

[0166] Through this update, the Critic network gradually optimizes the evaluation accuracy of each layout strategy, thereby providing more accurate feedback signals to the Actor network.

[0167] Step S28, Actor network parameter update: The Actor network optimizes its parameters according to the TD Error of the Critic network. If TD Error>0, it means that the current action is better than expected, and the Actor network will increase the probability of selecting the action; if TD Error<0, it means that the effect is lower than expected, and the Actor network will reduce the probability of selecting the action. Through gradual adjustment, the Actor network continuously optimizes the strategy in a multi-layout environment and improves the overall convergence efficiency.

[0168] Step S29, termination condition judgment: After each iteration, check whether the preset termination condition is met (such as TDError is less than or equal to the set threshold). When all layout strategies meet the termination condition, the training ends, and the final retrieval strategy is input into the precise box flipping value calculation module to ensure the accuracy of the model solution and the reliability of practical application.

[0169] Step S150, optimizing and solving the mathematical model through a commercial solver to obtain the container slot allocation plan and the terminal's maximum operating capacity for the current batch.

[0170] To further understand the present invention, Figure 8The overall framework of the solution to the new automated terminal slot allocation strategy provided by the present invention is demonstrated, and is explained as follows: a slot allocation model with uncertainty is constructed, and the operating characteristics of the single cantilever rail crane (URMGC) and the automatic guided vehicle (AGV) and the changes in the new yard layout are deeply considered to ensure that the slot allocation decision is highly consistent with the actual operating environment. At the same time, uncertain factors such as the order of picking up imported containers are included to more realistically reflect the challenges in actual operations; robust optimization is used to transform the uncertainty problem into a deterministic problem, and the original model is transformed into a deterministic problem by identifying and calculating the upper bound of the worst retrieval order. Transformed into a robust deterministic 0-1 integer programming model, so that slot allocation can be optimized under the most unfavorable conditions; all possible layouts within the bay are counted and classified, and homogeneous layout groups are summarized to reduce the complexity of the model and the number of constraints, thereby improving the efficiency of solution; the improved Actor-Critic deep reinforcement learning network is used to solve the specific value of the upper bound of the box flipping, and the data required for the mathematical model after the constraints are reduced; the model is solved by a commercial solver (or precise algorithm) to obtain the terminal's extreme operating capacity and optimal slot allocation plan under the current batch. This series of steps work together to provide a systematic process for solving the problem of uncertain slot allocation for imported containers, ensure the practical applicability and efficiency of the solution, and provide reliable technical support for terminal operators to efficiently manage terminal operations.

[0171] In summary, the present invention proposes an innovative slot allocation method, which combines deep reinforcement learning and robust optimization ideas to transform the uncertainty model into a deterministic model, thereby significantly improving the solution efficiency and real-time performance. In addition, by solving the import container allocation scheme under the most unfavorable retrieval order in the future, the terminal operator can not only reasonably optimize the slot allocation, but also effectively evaluate the terminal's container area operation capacity limit when handling the current import containers, thereby providing strong decision support for terminal operation management.

[0172] The present invention may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present invention.

[0173] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0174] Various aspects of the present invention are described herein with reference to the flow charts and / or block diagrams of the methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each box of the flow chart and / or block diagram and the combination of each box in the flow chart and / or block diagram can be implemented by computer-readable program instructions.

[0175] The flow charts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the systems, methods and computer program products according to multiple embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a part of a module, a program segment or an instruction, and a part of the module, a program segment or an instruction contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented by a dedicated hardware-based system that performs the specified function or action, or can be implemented by a combination of dedicated hardware and computer instructions. It is well known to those skilled in the art that it is equivalent to implement it by hardware, implement it by software, and implement it by combining software and hardware.

[0176] Embodiments of the present invention have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the marketplace, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein. The scope of the present invention is defined by the appended claims.

Claims

1. A method for allocating slots for automated terminal import containers embedded with deep reinforcement learning, comprising the following steps: For the container pickup area, an objective function is established based on the total processing time of the current batch of containers and the total time required to turn over all containers in the future; Performing robust equivalent transformation on the objective function to obtain a deterministic mathematical model; Using a deep reinforcement learning method to solve the upper bound of the objective function in the deterministic mathematical model; The optimization solution is obtained through commercial solver to obtain the slot allocation solution.

2. The method according to claim 1, characterized in that The objective function is set as: The constraints for solving the objective function are set as: in: i is the import box index, i∈{1,2,…,I}, I is the number of import boxes; j is the stack index, j∈{1,2,…,J}, J is the number of stack positions; k is the bay index, k∈{1,2,…,K}, K is the number of bays; e1 is the time it takes for the AGV to travel to the yard after loading the container from the shore crane; e2 is the time it takes for the AGV to return from the yard to the quay crane to wait for the container handover; n1 is the unit moving time of the AGV with the container; n2 is the unit moving time of the unloaded AGV; m1 is the unit horizontal movement time of the single-sided cantilever rail gantry crane between stacking positions; m2 is the unit horizontal movement time of the single-sided cantilever rail gantry crane between bays; H is the floor height; C is the capacity in bel positions; θ jk is the initial number of containers in stack position j, bay position k; r is the unit time of the box turning operation; x ijk is a decision variable. If the import box i is assigned to stack position j and bay position k, then x ijk =1, otherwise, x ijk =0; S i The import box i is assigned to the stack position; L i is the import box i is assigned to the bay position; T i is the total handling time of imported container i, which means the time from when container i waits for the AGV to pick up the container at the quay crane to when the AGV hands over the container to the single-side cantilever rail gantry crane; R(θ jk ;x ijk ) is the layout of the current bay k after the slot allocation, corresponding to the upper bound of the number of box turns required for future box pick-up; α and β are weighting coefficients of the corresponding terms.

3. The method according to claim 2, characterized in that The simplified objective function is set as: Compared with the objective function, the following constraints are added: in: τ is the layout index contained in the bay, τ∈{1,2,…,σ}, σ is the sum of all possible bay layouts; ω τ is the symbolic value of τ for different bay layouts; is the upper bound of the box flipping for different bay layouts τ; y τk is a decision variable. If the layout in bay k is τ, then y τk =1, otherwise, y τk =0.

4. The method according to claim 3, characterized in that The upper limit of the box turning Obtained through deep reinforcement learning, including the following steps: We use an actor-critic deep reinforcement learning approach to calculate the search order required to solve the upper bound of the exact bin flipping. Based on the calculated search order, a 0-1 integer programming model is used to obtain the exact solution of the search order under the set layout using a commercial solver as the upper bound of the box flipping.

5. The method according to claim 4, characterized in that By using the actor-critic deep reinforcement learning method, the search order required to solve the upper bound of the accurate box flipping is calculated, including the following steps: Set the initial state, where the current state of each bay layout is composed of the container arrangement order and layout identifier; The actor network outputs the probability distribution of actions based on the current state, and the selected actions include swap, reversal, or insertion operations; The environment module receives the action generated by the actor network and updates the current state to a new state Next State through the state transition module; The compact upper bound calculation module calculates the compact upper bound difference between the new and old states about box flipping as a reward value Reward, which reflects the immediate effect of the current action, and the compact upper bound calculation module uses the layout identifier to match the pre-stored layout information; The critic network estimates the value of the new state and the current state respectively, and obtains the value estimate of the new state and the value estimate of the current state; Calculate the time difference error reflecting the difference between the current state and the expected state as a feedback signal, and back-propagate the signal to the actor network and the critic network to adjust the parameters in the actor network and the critic network; When the time difference error meets the preset termination condition, the retrieval order learned at this time is output.

6. The method according to claim 5, characterized in that The time difference error is calculated according to the following formula: TD Error=Reward+γ*V(Next State)-V(Current State) Among them, TD Error represents the time difference error, Reward represents the reward value, V (Next State) is the value estimate of the new state, V (Current State) is the value estimate of the current state, and γ is the weighted coefficient of the value estimate of the new state.

7. A computer-readable storage medium having a computer program stored thereon, wherein: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

8. A computer device comprising a memory and a processor, wherein a computer program capable of running on the processor is stored in the memory, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

Images