Container terminal resource integrated scheduling method and system based on reinforcement learning and branch and price, and storage medium

Abstract

The application discloses a kind of based on reinforcement learning and branch pricing container terminal resource integrated scheduling method, system and storage medium.The method comprises: obtaining the scheduling associated data of container terminal scheduling, and constructing the original model of describing berth, shore crane and yard joint allocation problem based on scheduling associated data;Based on the block angle structure of original model, the original model is decomposed into a main problem and multiple pricing sub-problems, and the main problem and multiple pricing sub-problems after decomposition are solved based on the branch pricing framework constructed in advance to obtain linear relaxation optimal solution and integer feasible solution;The branching process of branch pricing framework is modeled as a tree Markov decision process, a tripartite graph representing the decision state is constructed based on the linking relationship of branching variable, main problem column and constraint node, and the target scheduling scheme is determined based on the tripartite graph and reinforcement learning dynamic branching strategy.The technical effects of guaranteeing the global optimality of scheduling scheme and meeting the complex scheduling requirements of real terminal are achieved.

Description

Technical Field

[0001] This invention relates to the field of port scheduling technology, and in particular to a container terminal resource integrated scheduling method, system and storage medium based on reinforcement learning and branch pricing. Background Technology

[0002] The key to improving the operational efficiency of container terminals lies in the rational allocation and utilization of core resources such as berths, quay cranes, and yard space. Berth allocation, quay crane allocation, and yard allocation are interdependent, necessitating consideration of their integrated approach (BQCYAP). On one hand, berth allocation must consider the time vessels spend using the berths, which primarily depends on the vessel's total loading / unloading volume and the number of quay cranes assigned to it. On the other hand, the total horizontal transport route length for each vessel is determined by both the assigned berth and the yard location.

[0003] The existing technology has the following main drawbacks:

[0004] Most algorithms for solving the BQCYAP problem are currently heuristic or metaheuristic. While these algorithms can provide feasible solutions within an acceptable timeframe, they cannot guarantee optimality, and the quality of their solutions often deviates significantly from the theoretically optimal solution, resulting in unmaximized resource utilization. Therefore, there is an urgent need for a method that can efficiently and accurately solve the integrated optimization problem of container terminal berths, quay cranes, and yards to meet the ever-increasing demands of terminal operations. Summary of the Invention

[0005] This invention provides a container terminal resource integration scheduling method, system, and storage medium based on reinforcement learning and branch pricing, to achieve an efficient and accurate solution to the integrated optimization problem of container terminal berths, quay cranes, and yards, in order to meet the ever-increasing demands of terminal operations.

[0006] According to one aspect of the present invention, a container terminal resource integrated scheduling method based on reinforcement learning and branch pricing is provided, the method comprising:

[0007] Obtain scheduling correlation data for container terminal scheduling, and construct an original model describing the joint allocation problem of berths, quay cranes and yards based on the scheduling correlation data; wherein, the scheduling correlation data includes basic ship data, terminal resource data and cost rule data, and the original model aims to minimize the total operating cost and includes resource and time-related constraints;

[0008] Based on the block structure of the original model, the original model is decomposed into a main problem and multiple pricing sub-problems. The decomposed main problem and multiple pricing sub-problems are solved based on a pre-constructed branch pricing framework to obtain the linear relaxation optimal solution and integer feasible solution.

[0009] The branching process of the branch pricing framework is modeled as a tree Markov decision process. A three-part graph representing the decision state is constructed based on the linking relationships of branch variables, main problem series, and constraint nodes. The target scheduling scheme is determined based on the three-part graph and the reinforcement learning dynamic branching strategy.

[0010] According to another aspect of the present invention, a container terminal resource integration scheduling system based on reinforcement learning and branch pricing is provided, the system comprising:

[0011] The original model building module is used to acquire scheduling correlation data of container terminal scheduling, and to build an original model describing the joint allocation problem of berths, quay cranes and yards based on the scheduling correlation data; wherein, the scheduling correlation data includes basic ship data, terminal resource data and cost rule data, and the original model aims to minimize the total operating cost and includes resource and time-related constraints;

[0012] The model decomposition module is used to decompose the original model into a main problem and multiple pricing sub-problems based on the block structure of the original model. The decomposed main problem and multiple pricing sub-problems are solved based on a pre-built branch pricing framework to obtain the linear relaxation optimal solution and integer feasible solution.

[0013] The scheduling scheme determination module is used to model the branching process of the branch pricing framework as a tree Markov decision process, construct a three-part graph representing the decision state based on the linking relationship of branch variables, main problem series and constraint nodes, and determine the target scheduling scheme based on the three-part graph and reinforcement learning dynamic branching strategy.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] At least one processor;

[0016] and memory that is communicatively connected to at least one processor;

[0017] The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to execute the container terminal resource integration scheduling method based on reinforcement learning and branch pricing according to any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the container terminal resource integration scheduling method based on reinforcement learning and branch pricing according to any embodiment of the present invention.

[0019] The technical solution of this invention involves acquiring scheduling correlation data of container terminal scheduling, and constructing an original model describing the joint allocation problem of berths, quay cranes, and yards based on the scheduling correlation data. The scheduling correlation data includes basic ship data, terminal resource data, and cost rule data. The original model aims to minimize total operating costs and includes resource and time-related constraints. Based on the block structure of the original model, it is decomposed into a main problem and multiple pricing sub-problems. A pre-constructed branch pricing framework is used to solve the decomposed main problem and multiple pricing sub-problems to obtain a linear... The optimal solution and integer feasible solution are relaxed. The branching process of the branch pricing framework is modeled as a tree Markov decision process. A three-part graph representing the decision state is constructed based on the linking relationship between branch variables, main problem series and constraint nodes. The target scheduling scheme is determined based on the three-part graph and reinforcement learning dynamic branching strategy. This solves the technical problem of difficulty in balancing the optimality of the solution and the solution efficiency in the existing integrated scheduling of container terminal resources. It realizes accurate solution and intelligent branching decision for large-scale joint scheduling of berths, quay cranes and yards. It achieves the technical effects of ensuring the global optimality of the scheduling scheme, significantly improving the solution efficiency and meeting the complex scheduling needs of real terminals.

[0020] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 A flowchart illustrating a container terminal resource integration scheduling method based on reinforcement learning and branch pricing, provided as an embodiment of the present invention;

[0023] Figure 2a A flowchart illustrating another container terminal resource integration scheduling method based on reinforcement learning and branch pricing, provided in an embodiment of the present invention;

[0024] Figure 2b A schematic diagram of a dynamic branching strategy framework for an optional example of a container terminal resource integration scheduling method based on reinforcement learning and branch pricing, provided in an embodiment of the present invention.

[0025] Figure 2cA three-part diagram illustrating an optional example of a container terminal resource integration scheduling method based on reinforcement learning and branch pricing, provided by an embodiment of the present invention.

[0026] Figure 3 This is a schematic diagram of the structure of a container terminal resource integration scheduling system based on reinforcement learning and branch pricing, provided in an embodiment of the present invention.

[0027] Figure 4 A schematic diagram of the structure of an electronic device for implementing a container terminal resource integration scheduling method based on reinforcement learning and branch pricing, according to an embodiment of the present invention. Detailed Implementation

[0028] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0030] Figure 1 This is a flowchart illustrating a container terminal resource integration scheduling method based on reinforcement learning and branch pricing, provided as an embodiment of the present invention. This embodiment is applicable to port container terminal resource scheduling. The method can be executed by a container terminal resource integration scheduling system based on reinforcement learning and branch pricing. This system can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method specifically includes the following steps:

[0031] S110. Obtain scheduling correlation data for container terminal scheduling, and construct an original model describing the joint allocation problem of berths, quay cranes and yards based on the scheduling correlation data; wherein, the scheduling correlation data includes basic ship data, terminal resource data and cost rule data, and the original model aims to minimize the total operating cost and includes resource and time-related constraints.

[0032] Among these, the scheduling correlation data can be the core dataset supporting resource scheduling decisions at container terminals. The original model can be a mathematical model built upon the scheduling correlation data.

[0033] Optionally, basic vessel data includes the number of vessels, the expected berthing time window for each vessel, the feasible berthing time window, the departure time (time window), the volume of containers loaded and unloaded (the number of imported or exported containers), and the company to which the vessel belongs; terminal resource data includes information on the terminal's available resources, such as the number and location of berths, the total number of quay cranes and the operational efficiency of different configurations, the number of yard sub-blocks and the companies to which they belong, and the scheduling cycle and time segment division; cost rule data includes rule data for calculating scheduling costs and constraining scheduling behavior, such as the time penalty cost coefficient for early arrival or late departure of vessels, the unit transportation distance cost coefficient for containers, and resource constraint rules such as no duplicate occupation of berths.

[0034] Specifically, three types of scheduling-related data are collected: ship basics, terminal resources, and cost rules. Based on the scheduling-related data, an original model is constructed to clarify the decision variables for the joint allocation of berths, quay cranes, and yards. With the goal of minimizing total operating costs, resource (e.g., berth uniqueness) and time constraints are set.

[0035] Optionally, the original model is a mixed-integer programming model; the resource and time-related constraints include at least one of the following:

[0036] The constraint that each vessel is allocated only one berth;

[0037] Conflict constraints to prevent duplicate allocation of berths within the same time period;

[0038] Capacity constraints that limit the total number of quay cranes in use;

[0039] Time constraints that the vessel's operation time falls within the expected service time window;

[0040] The vessel is designated to use the sub-block of its own company's yard, subject to ownership constraints.

[0041] Among these, the mixed-integer programming model can be understood as the mathematical type of the original model. Conflict constraints can be understood as constraints on berth usage time. Capacity constraints can be understood as constraints on the number of quay cranes. Time constraints can be understood as constraints on ship operation time. Ownership constraints can be understood as constraints on yard usage.

[0042] Specifically, the original model is a mixed integer programming model, which determines that the decision variables include integer variables (such as berth allocation variables) and continuous variables (such as operation time variables) to match the mathematical characteristics of the scheduling problem.

[0043] For example, five types of constraints—uniqueness, conflict, capacity, time, and attribution—are used as constraints in the original model to ensure the legality of the scheduling scheme. For example, a uniqueness constraint of "one berth per ship" and a capacity constraint of "total number of quay cranes not exceeding the limit" are set.

[0044] S120. Based on the block structure of the original model, the original model is decomposed into a main problem and multiple pricing sub-problems. The decomposed main problem and multiple pricing sub-problems are solved based on the pre-constructed branch pricing framework to obtain the linear relaxation optimal solution and the integer feasible solution.

[0045] In this context, the block structure can be understood as the constraint characteristics of the original model. The main problem can be understood as the sub-model responsible for global coordination after model decomposition. For example, selecting a combination from the single-ship scheduling plans of multiple ships to ensure that global resource constraints such as berths, quay cranes, and yards are met, with the goal of minimizing the total combined cost. The pricing sub-problem can be understood as the sub-model corresponding to a single ship after model decomposition. For example, generating a new scheduling plan with the lowest cost for each ship. The branch pricing framework (BP framework) can be understood as the BP (Branch and Price) framework. The linear relaxation optimal solution can be understood as the theoretical optimal solution obtained in the BP framework after relaxing the constraint that the original model variables are integers. The integer feasible solution can be understood as a feasible scheduling scheme that satisfies the condition that all variables are integers.

[0046] Specifically, utilizing the block structure of the original model, it is broken down into one main problem and multiple pricing sub-problems. The main and sub-problems are then solved iteratively using a backpropagation (BP) framework to obtain the theoretically optimal linear relaxation solution and a feasible integer solution. The main problem is used to coordinate global resources, while the pricing sub-problems are used to generate the optimal plan for a single ship.

[0047] Optionally, the step of solving the decomposed main problem and multiple pricing sub-problems based on a pre-built branch pricing framework to obtain a linearly relaxed optimal solution and an integer feasible solution includes: iteratively solving the decomposed main problem and pricing sub-problems based on a pre-built branch pricing framework; obtaining the linearly relaxed optimal solution through a column generation mechanism; and generating an integer feasible solution by combining a master heuristic algorithm.

[0048] The column generation mechanism can be understood as generating new single-ship scheduling plans, i.e., columns, from the pricing subproblem. The master heuristic algorithm can be understood as an algorithm that quickly generates integer feasible solutions from non-integer solutions.

[0049] Specifically, within the BP framework, the initial solution to the main problem is first solved to obtain the resource dual price. The dual price is then substituted into the pricing subproblem to generate a new single-ship scheduling plan. If the new plan reduces the cost of the main problem, it is added to the main problem, and this process is repeated. Iteration is performed through a column generation mechanism until no new plan can be added, yielding the optimal linear relaxation solution to the main problem. Simultaneously, the main heuristic algorithm is invoked to fix the non-integer variables in the linear relaxation solution to integers, and after verifying the constraints, an integer feasible solution is generated.

[0050] S130. The branching process of the branch pricing framework is modeled as a tree Markov decision process. A three-part graph representing the decision state is constructed based on the linking relationship between branch variables, main problem series and constraint nodes. The target scheduling scheme is determined based on the three-part graph and the reinforcement learning dynamic branching strategy.

[0051] Among them, Tree Markov Decision Process (Tree MDP) can be understood as modeling the branching process of the BP framework as a state-action-reward decision model. The ternary graph can be understood as a structured graph representing the branching decision states.

[0052] Reinforcement learning dynamic branching strategies can be understood as branch decision-making methods based on reinforcement learning. Objective scheduling schemes can be understood as the final executable scheduling results.

[0053] Specifically, the branching process of the BP framework is modeled as a Tree MDP, and a three-part graph is constructed based on the association between branch variables, main problem columns, and constraint nodes. Combined with reinforcement learning dynamic branching strategies, the optimal branch decision is identified from the three-part graph, and the target scheduling scheme is finally determined.

[0054] Optionally, determining the target scheduling scheme based on the tripartite graph and the reinforcement learning dynamic branching strategy includes: inputting the tripartite graph into a pre-trained graph convolutional neural network to obtain the optimal branch decision of the model output; embedding the optimal branch decision into the branch pricing framework, and determining the target scheduling scheme that satisfies global optimality through branch bounding and pruning mechanisms.

[0055] Graph Convolutional Neural Networks (GCNs) can be understood as neural networks that process graph-structured data. Optimal branch decision can be understood as selecting the variable from candidate branch variables that best narrows the search tree and accelerates the solution during the branching phase of the backpropagation (BP) framework. Branch and bound and pruning mechanisms can be understood as the search mechanism of the BP framework. Global optimality can be understood as the criterion for proving the optimality of the target scheduling scheme; for example, the lower bound of cost equals the upper bound of cost, with a gap of 0.00%, ensuring that the scheme is the optimal solution globally.

[0056] Specifically, the tripartite graph representing the current decision state is input into a pre-trained GCN. The GCN analyzes the correlation information between branch variables, the main problem sequence, and constraint nodes through graph convolutional layers, outputting the selection probability of each candidate branch variable, and finally determining the optimal branch decision. The optimal branch decision is integrated into the BP framework, adding new constraints to the current search tree node and generating child nodes; the upper and lower bounds of the cost of child nodes are calculated through branch and bound, and nodes exceeding the global upper bound are pruned until all nodes have been explored, resulting in a target scheduling scheme that satisfies global optimality.

[0057] Optionally, by inputting the ternary graph into a pre-trained graph convolutional neural network to obtain the optimal branch decision output by the model, the following steps are taken: inputting the ternary graph into a pre-trained graph convolutional neural network, aggregating the correlation information between branch variables, main problem columns and constraint nodes through graph convolutional layers, and outputting the selection probability of each candidate branch variable; selecting the candidate branch variable with the highest selection probability as the optimal branch decision based on multiple selection probabilities.

[0058] The graph convolutional layer can be understood as the core layer of GCN, capturing the relationships between candidate branch variable nodes, main problem column nodes, and main problem constraint nodes in the tripartite graph, and transforming these relationships into node features. Candidate branch variables can be understood as variables that can be used for branching in the BP framework. Examples include whether ship A is assigned to berth B and whether ship A uses quay crane configuration C. The probability of selection can be understood as a quantified value of the degree of recommendation given by GCN to each candidate branch variable.

[0059] Specifically, the tripartite graph is input into the GCN, and the graph convolutional layer automatically aggregates the association information of the three types of nodes. The network then calculates and outputs the selection probability of each candidate branch variable. By comparing the selection probabilities of all candidate branch variables, the variable with the highest probability is selected as the final optimal branch decision.

[0060] Optionally, the target scheduling scheme includes berth allocation results, quay crane configuration results, yard assignment results, and operation time arrangement results for each vessel.

[0061] The berth allocation result can be understood as the specific berth assigned to each vessel. The quay crane configuration result can be understood as the type and number of quay cranes used by each vessel. The yard assignment result can be understood as the yard sub-block assigned to each vessel. The operation time scheduling result can be understood as the start or end time of operations for each vessel.

[0062] Specifically, from the final global optimal solution of the BP framework, four types of core decision-making information for each vessel are extracted: the corresponding berth, the configuration of the quay crane used, the assigned yard sub-block, and the start and end times of the operation. These four types of information are then organized for each vessel to form a structured target scheduling plan, which can directly guide actual terminal operations.

[0063] The technical solution of this invention involves acquiring scheduling correlation data of container terminal scheduling, and constructing an original model describing the joint allocation problem of berths, quay cranes, and yards based on the scheduling correlation data. The scheduling correlation data includes basic ship data, terminal resource data, and cost rule data. The original model aims to minimize total operating costs and includes resource and time-related constraints. Based on the block structure of the original model, it is decomposed into a main problem and multiple pricing sub-problems. A pre-constructed branch pricing framework is used to solve the decomposed main problem and multiple pricing sub-problems to obtain a linear... The optimal solution and integer feasible solution are relaxed. The branching process of the branch pricing framework is modeled as a tree Markov decision process. A three-part graph representing the decision state is constructed based on the linking relationship between branch variables, main problem series and constraint nodes. The target scheduling scheme is determined based on the three-part graph and reinforcement learning dynamic branching strategy. This solves the technical problem of difficulty in balancing the optimality of the solution and the solution efficiency in the existing integrated scheduling of container terminal resources. It realizes accurate solution and intelligent branching decision for large-scale joint scheduling of berths, quay cranes and yards. It achieves the technical effects of ensuring the global optimality of the scheduling scheme, significantly improving the solution efficiency and meeting the complex scheduling needs of real terminals.

[0064] Figure 2a This is a flowchart illustrating another container terminal resource integration and scheduling method based on reinforcement learning and branch pricing, provided as an embodiment of the present invention. This embodiment is a further refinement of the above embodiments. Specific implementation details can be found in the technical solution of this embodiment. Technical terms that are the same as or corresponding to those in the above embodiments will not be repeated here.

[0065] like Figure 2a As shown, the method specifically includes the following steps:

[0066] S210. Obtain scheduling correlation data for container terminal scheduling, and construct an original model describing the joint allocation problem of berths, quay cranes and yards based on the scheduling correlation data; wherein, the scheduling correlation data includes basic ship data, terminal resource data and cost rule data, and the original model aims to minimize the total operating cost and includes resource and time-related constraints.

[0067] S220. Based on the block structure of the original model, the original model is decomposed into a main problem and multiple pricing sub-problems. The decomposed main problem and multiple pricing sub-problems are solved based on the pre-constructed branch pricing framework to obtain the linear relaxation optimal solution and the integer feasible solution.

[0068] S230. Model the branching process of the branch pricing framework as a tree Markov decision process, and determine the candidate branch variable nodes, the main problem column nodes, and the main problem constraint nodes.

[0069] In this diagram, candidate branch variable nodes can be understood as nodes representing branchable variables in the ternary graph. Main problem column nodes can be understood as nodes representing existing single-ship scheduling plans in the main problem within the ternary graph; for example, each node corresponds to a berth, quay crane, and yard combination scheme for a single ship. Main problem constraint nodes can be understood as nodes representing global constraints of the main problem in the ternary graph; for example, berth non-duplication constraints and quay crane capacity constraints.

[0070] Specifically, from the information of the current node in the BP framework, candidate branch variables are extracted as candidate branch variable nodes; existing single-ship scheduling plans in the main problem are extracted as main problem column nodes; and global constraints of the main problem are extracted as main problem constraint nodes.

[0071] S240. Add features to the candidate branch variable node, the main problem column node, and the main problem constraint node, respectively.

[0072] Among them, node features can be understood as numbers used to describe node attributes.

[0073] Specifically, additional descriptive attributes are added to the three types of nodes to make the node information more complete.

[0074] Optionally, features are added to the candidate branch variable nodes, the main problem column nodes, and the main problem constraint nodes, including: adding features such as variable solution values, corresponding ship and resource types, and historical branch counts to the candidate branch variable nodes; adding features such as column costs, test numbers, and main variable values ​​to the main problem column nodes; and adding features such as dual optimal values ​​and constraint corresponding resource types to the main problem constraint nodes.

[0075] In this context, the variable solution value can be understood as the value of the candidate branch variable in the current linear relaxation solution. Ship and resource type can be understood as the specific object associated with the candidate branch variable. For example, ship A and berth B, or ship A + quay crane configuration C. Historical branch count can be understood as the number of times the candidate branch variable was selected in past branching processes. Column cost can be understood as the total cost of the single-ship scheduling plan corresponding to the column node of the main problem, including the time penalty cost and transportation cost of the plan. Test number can be understood as an indicator of the quality of the main problem column. The main variable value can be understood as the variable value used to select a specific single-ship plan in the main problem. The dual optimal value can be understood as the value of the dual variable corresponding to the linear relaxation solution of the main problem. The constraint corresponding to the resource type can be understood as the resource category associated with the constraint of the main problem, for example, berth resources and quay crane resources.

[0076] Specifically, for each candidate branch variable node, the current solution value of the variable, the associated ship and resource types, and the number of times it has been branched in the past are added to quantify the variable's state and history. For each main problem column node, the cost, test number, and main variable values ​​of the corresponding single-ship plan are added to reflect the plan's cost advantage and its role in the main problem. For each main problem constraint node, the dual optimal value and the associated resource type are added to reflect the shadow price and control object of the constraint.

[0077] S250. Build links between nodes after adding features to obtain a tripartite graph representing the decision state, and determine the target scheduling scheme based on the tripartite graph and the reinforcement learning dynamic branching strategy.

[0078] In this context, the links between nodes can be understood as the edges that connect different types of nodes in a tripartite graph.

[0079] Specifically, based on the actual relationships between nodes, such as the main problem column containing branch variables or the main problem column satisfying constraints, links are established between corresponding nodes to form a complete ternary graph.

[0080] The technical solution of this invention, by identifying three types of core nodes, supplementing node features, and constructing links between nodes to form a tripartite graph, can accurately and comprehensively represent the decision state under the branch pricing framework. This provides structured information support for reinforcement learning agents to clearly understand branch scenarios and output efficient branch decisions, thereby improving the accuracy of branch decisions and the solution efficiency of branch pricing algorithms.

[0081] As an optional example of Embodiment 1 of the present invention, the container terminal resource integrated scheduling method based on reinforcement learning and branch pricing in this embodiment specifically includes the following steps:

[0082] Step 1: Based on the terminal operation scenario, establish a mixed-integer programming primal model (OP) describing the joint allocation problem of berths, quay cranes, and yards. The main decision variables of this model include: berth allocation, quay crane configuration selection, yard block assignment, and the start time of each vessel's operations. Its optimization objective is to minimize the total operating cost, including time window penalties and transportation distance. The model considers multiple constraints such as resource uniqueness, operational logic, resource capacity, and time windows.

[0083] Step 2: Utilizing the block structure of the original model (OP), where most constraints are only related to a single ship and a few are globally coupled constraints, apply Dantzig-Wolfe decomposition. The original problem is decomposed into a high-level master problem (MP) and multiple parallel pricing subproblems (PP). The master problem acts as a coordinator, responsible for combining and selecting from numerous feasible "complete single-ship scheduling plans" to satisfy global resource constraints; each pricing subproblem is responsible for generating the new, lowest-cost, feasible scheduling plan for a single ship.

[0084] Step 3: Construct the branch-and-price (BP) algorithm framework to solve the model decomposed in Step 2. This framework obtains the optimal solution for linear relaxation at each node of the search tree by iteratively solving the main problem and the pricing subproblem (i.e., column generation), and combines it with the master heuristic algorithm to find high-quality integer feasible solutions, thereby achieving an efficient search of the solution space.

[0085] Step 4: To address the efficiency bottleneck of the BP algorithm in branch decision-making, a dynamic branching strategy based on reinforcement learning is designed. Figure 2b A schematic diagram of a dynamic branching strategy framework for an optional instance of a container terminal resource integration scheduling method based on reinforcement learning and branch pricing is provided. Figure 2b As shown, this strategy models the branching process as a Tree Markov Decision Process (Tree MDP) and proposes a novel three-part graph state representation method to describe the decision state. Finally, it outputs the optimal branch decision through a graph convolutional neural network (GCN) policy network. Figure 2c A three-part diagram illustrating an optional instance of a container terminal resource integration scheduling method based on reinforcement learning and branch pricing is provided. (See diagram below.) Figure 2c As shown, this ternary graph contains three types of nodes: candidate branch variables, main problem columns, and main problem constraints, along with their rich feature information.

[0086] Step 5: Embed the dynamic branching strategy designed in Step 4 into the BP framework of Step 3 to solve specific problem instances online. The GCN policy network needs to be trained offline through reinforcement learning (such as the PPO algorithm) on multiple training instances to enable it to generalize and make efficient decisions.

[0087] Furthermore, the original model (OP) established in step one is as follows:

[0088]

[0089] st

[0090]

[0091]

[0092]

[0093] Please refer to Tables 1 and 2 for the explanation of the mathematical symbols.

[0094]

[0095] Table 1

[0096]

[0097] Table 2

[0098] Formula (1) represents minimizing the total cost, including time penalty costs and transportation costs. This represents the penalty cost for the deviation between the planned berthing window and the expected berthing window, while represents the export container transportation cost. This represents the cost of transporting imported containers. Constraint (2) means that each ship will be assigned to a specific berth. Constraint (3) stipulates that each ship has one and only one quay crane configuration selected. Constraint (4) ensures that a given number of sub-blocks are reserved for each ship. Constraint (5) restricts each ship to using only its own company's yard area. Constraint (6) stipulates that each ship must begin loading and unloading within a certain time step of the cycle. Constraint (7) utilizes Define the start time of loading and unloading of the ship Constraint (8) defines the ship's final unloading time. Constraints (9) and (10) mandate that the loading and unloading times of each ship must fall within its feasible time window. Constraints (11) and (12) respectively advance the berthing time. Departure time delay Linearization to and Constraints (13) to (15) will affect the decision variables. and The result of the AND operation is linearized as follows: Constraints (16) to (18) will affect the decision variables. and The result of the AND operation is linearized as follows: Constraints (19) to (23) define the decision variables. Constraints (24)-(26) will bind the decision variables. and The result of the AND operation is linearized as follows: Constraint (27) ensures that the same berth can be allocated to at most one ship in the same time period. Constraint (28) calculates the number of quay cranes used in each time period. Constraint (29) ensures that the number of quay cranes used in each time step cannot exceed the actual available number. Constraint (30) ensures that each yard sub-block can be allocated to at most one ship. The range of values ​​for all variables is defined by (31)-(32). Further, in step two, the objective function of the decomposed main problem (MP) is shown in formula (30), and its constraints include convexity constraints (such as formula (31)) and global resource sharing constraints (such as formulas (32), (33), and (34)). The objective function of the pricing sub-problem (PP) is shown in formula (37).

[0099] Furthermore, in step 2, the decomposed master problem (MP) model is as follows:

[0100]

[0101] st

[0102]

[0103] in, Indicates according to Allocation plan, container ships exist Whether to use berths during the time period . Indicates according to Allocation plan storage yard sub-blocks Whether or not it is assigned to a container ship . Indicates according to Allocation plan, time period Time allocation to container ships The number of quay bridges; Indicates plan In the above formula, the optimization objective of OP is rewritten as Equation (33). Equation (34) ensures that each ship in the solution has a feasible allocation. Equation (35) guarantees that each berth can be occupied by at most one ship in each time step. Equation (36) means that each sub-block can be reserved for at most one ship. Equation (37) stipulates that the quay cranes used in each time period are within the finite capacity. Equation (38) defines the range of values ​​for the decision variables.

[0104] Furthermore, in step 3, the BP solution framework specifically includes:

[0105] Column generation: At each node of the branch and bound search tree, by iteratively solving the linear relaxation and pricing subproblem (PP) of the restricted master problem (RMP), a new feasible scheduling plan (column) is dynamically generated for the master problem until no new column with a negative test number can be found, thus obtaining the optimal solution of linear relaxation for that node.

[0106] Branching and Searching: When the linear relaxation solution of a node is not an integer, child nodes are created according to specific branching rules (determined in step four), and the search tree is explored according to a certain node selection strategy (such as depth-first search). Each node here represents a subset of the original domain. Branching a node means adding two constraints to the current subset, dividing it into three parts. One part does not contain integer solutions and is therefore discarded, while the other two parts form two new child nodes. Exploring child nodes means finding linear relaxation solutions on the subset represented by this child node. If the solution is still not an integer, branching continues.

[0107] During the solution process, various master heuristic algorithms (such as the fixed heuristic based on RMP fractional solutions) are employed to attempt to quickly construct high-quality integer feasible solutions from the current non-integer solutions, in order to update the global upper bound column generation as early as possible: by iteratively solving the linear relaxation and pricing subproblems (PP) of the restricted master problem (RMP), new feasible scheduling plans (columns) are dynamically generated for the master problem until no new column with a negative test number can be found. A feasible scheduling plan can be understood as a detailed "resource scheduling scheme." For example, allocating a resource to a specific ship at a given time.

[0108] Furthermore, in step 4, the dynamic branching strategy based on reinforcement learning specifically includes:

[0109] Tree MDP Modeling: The search tree exploration process of BP is modeled as Tree MDP, and its reward function only considers the impact of the current branch action on its subtree, thereby achieving more accurate credit allocation.

[0110] Tripartite graph state representation: To enable reinforcement learning agents to fully understand the node states under the BP framework, a tripartite graph consisting of candidate branch variable nodes, main problem column nodes, and main problem constraint nodes is proposed to represent the current state.

[0111] Furthermore, the tripartite diagram represents... All the information that constitutes it comes from the column generation process. In a tripartite graph... These are, respectively, the set of candidate branch variable endpoints, the set of selected column endpoints, and the set of constraint endpoints for a node to be branched. This represents the set of edges between the selected column set and the other two types of nodes. The design principles for each endpoint, feature encoding, and edge weight are as follows:

[0112] For each candidate branch variable endpoint Its encoding It must include the solution value of the variable, the type of the ship and resource entity it points to, the number of historical selections, and the resulting average lower bound improvement, among other characteristics.

[0113] For each column endpoint set Its encoding It must include the main variable. , Cost Features such as column reduced cost, etc.

[0114] For each constraint endpoint Its encoding It must include the dual optimal value ( , , , The constraints specify characteristics such as the types of ships and resource entities to which they apply.

[0115] For each column endpoint and the candidate branch variable endpoint... The weights represent the contributions of adjacent columns to the candidate branch variable. For example, ,one Make Then the weight of the edge is or .

[0116] For each column endpoint and constraint endpoint edge The weight represents the contribution of adjacent columns to the left side of the constraint. For example, column Constraints The contribution on the right is Therefore, the weight of the edge is .

[0117] In addition, the state features of the entire search tree This is necessary; it includes the current global upper bound, the average distribution of explored nodes, and the positions of candidate nodes. Concatenating features to the end of each node would introduce a significantly larger number of parameters. Therefore, the proposed method only considers concatenating features later in the forward propagation. Integrate with node information.

[0118] A Graph Convolutional Neural Network (GCN) is used to parameterize the policy and value function. This network takes a tripartite graph as input, aggregates information through graph convolutional layers, and ultimately outputs the probability of selecting each candidate branch variable. In reinforcement learning, our policy is described as a probability distribution function. During training, to encourage trying more branch options, sampling is performed according to probability (meaning low-probability actions may also be adopted). In actual use, the branch action with the highest probability is selected.

[0119] To accelerate training, the GCN policy network is warm-started using prior knowledge of the hierarchical branching strategy, thereby improving the convergence speed in the early stages of training.

[0120] The technical solution of this invention is based on the precise algorithm framework of branch pricing. By decomposing the original problem into a main problem and sub-problems and solving them iteratively, the optimality of the solution is guaranteed. At the same time, at the key decision points of the branch and bound search tree, an intelligent branching strategy based on reinforcement learning and graph neural networks is innovatively introduced, which greatly improves the solution efficiency of the algorithm and enables it to efficiently provide precise optimal solutions for the joint scheduling problem of large-scale container terminal resources.

[0121] Figure 3 This is a schematic diagram of a container terminal resource integration scheduling system based on reinforcement learning and branch pricing, provided as an embodiment of the present invention. Figure 3 As shown, the system includes: an original model construction module 310, a model decomposition module 320, and a scheduling scheme determination module 330.

[0122] The original model construction module 310 is used to acquire scheduling-related data of container terminal scheduling and construct an original model describing the joint allocation problem of berths, quay cranes and yards based on the scheduling-related data. The scheduling-related data includes basic ship data, terminal resource data and cost rule data. The original model aims to minimize the total operating cost and includes resource and time-related constraints. The model decomposition module 320 is used to decompose the original model into a main problem and multiple pricing sub-problems based on the block structure of the original model. The decomposed main problem and multiple pricing sub-problems are solved based on a pre-constructed branch pricing framework to obtain the linear relaxation optimal solution and integer feasible solution. The scheduling scheme determination module 330 is used to model the branching process of the branch pricing framework as a tree Markov decision process. A three-part graph representing the decision state is constructed based on the link relationship of branch variables, main problem columns and constraint nodes. The target scheduling scheme is determined based on the three-part graph and reinforcement learning dynamic branching strategy.

[0123] The technical solution of this invention involves acquiring scheduling correlation data of container terminal scheduling, and constructing an original model describing the joint allocation problem of berths, quay cranes, and yards based on the scheduling correlation data. The scheduling correlation data includes basic ship data, terminal resource data, and cost rule data. The original model aims to minimize total operating costs and includes resource and time-related constraints. Based on the block structure of the original model, it is decomposed into a main problem and multiple pricing sub-problems. A pre-constructed branch pricing framework is used to solve the decomposed main problem and multiple pricing sub-problems to obtain a linear... The optimal solution and integer feasible solution are relaxed. The branching process of the branch pricing framework is modeled as a tree Markov decision process. A three-part graph representing the decision state is constructed based on the linking relationship between branch variables, main problem series and constraint nodes. The target scheduling scheme is determined based on the three-part graph and reinforcement learning dynamic branching strategy. This solves the technical problem of difficulty in balancing the optimality of the solution and the solution efficiency in the existing integrated scheduling of container terminal resources. It realizes accurate solution and intelligent branching decision for large-scale joint scheduling of berths, quay cranes and yards. It achieves the technical effects of ensuring the global optimality of the scheduling scheme, significantly improving the solution efficiency and meeting the complex scheduling needs of real terminals.

[0124] Optionally, the scheduling scheme determination module includes:

[0125] The optimal branch decision acquisition unit is used to obtain the optimal branch decision of the model output by inputting the three-part graph into a pre-trained graph convolutional neural network.

[0126] The scheduling scheme determination unit is used to embed the optimal branch decision into the branch pricing framework and determine the target scheduling scheme that satisfies global optimality through branch delimitation and pruning mechanisms.

[0127] Optionally, the optimal branch decision acquisition unit includes:

[0128] The selection probability output subunit is used to input the three-part graph into a pre-trained graph convolutional neural network, aggregate the correlation information between branch variables, main problem columns and constraint nodes through graph convolutional layers, and output the selection probability of each candidate branch variable.

[0129] The optimal branch decision determination sub-unit is used to select the candidate branch variable with the highest selection probability as the optimal branch decision based on multiple selection probabilities.

[0130] Optionally, the model decomposition module includes:

[0131] Iterative solution unit, used to iteratively solve the decomposed main problem and pricing subproblems based on a pre-built branch pricing framework;

[0132] The solution acquisition unit is used to obtain the optimal solution of linear relaxation through the column generation mechanism and to generate an integer feasible solution in combination with the master heuristic algorithm.

[0133] Optionally, the scheduling scheme determination module includes:

[0134] The node determination unit is used to determine candidate branch variable nodes, main problem column nodes, and main problem constraint nodes;

[0135] The feature addition unit is used to add features to the candidate branch variable node, the main problem column node, and the main problem constraint node, respectively.

[0136] The ternary graph construction unit is used to construct links between nodes after adding features, so as to obtain the ternary graph representing the decision state.

[0137] Optionally, the feature adding unit includes:

[0138] The first feature addition subunit is used to add features such as variable solution value, corresponding ship and resource type and historical branch number to the candidate branch variable node;

[0139] The second feature adds a sub-unit, which is used to add column cost and test number main variable values ​​to the main problem column node;

[0140] The third feature addition subunit is used to add features of dual optimal values ​​and resource types corresponding to constraints to the main problem constraint nodes.

[0141] Optionally, the original model is a mixed-integer programming model; the resource and time-related constraints include at least one of the following:

[0142] The constraint that each vessel is allocated only one berth;

[0143] Conflict constraints to prevent duplicate allocation of berths within the same time period;

[0144] Capacity constraints that limit the total number of quay cranes in use;

[0145] Time constraints that the vessel's operation time falls within the expected service time window;

[0146] The vessel is designated to use the sub-block of its own company's yard, subject to ownership constraints.

[0147] Optionally, the target scheduling scheme includes berth allocation results, quay crane configuration results, yard assignment results, and operation time arrangement results for each vessel.

[0148] The container terminal resource integration scheduling system based on reinforcement learning and branch pricing provided in this embodiment of the invention can execute the container terminal resource integration scheduling method based on reinforcement learning and branch pricing provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0149] Figure 4 This is a schematic diagram of an electronic device for implementing the container terminal resource integration scheduling method based on reinforcement learning and branch pricing, as described in this embodiment of the invention. The electronic device is intended to represent various forms of digital computers, such as laptops, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0150] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0151] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0152] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the method of integrated scheduling of container terminal resources based on reinforcement learning and branch pricing.

[0153] In some embodiments, the method for integrated scheduling of container terminal resources based on reinforcement learning and branch pricing can be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method for integrated scheduling of container terminal resources based on reinforcement learning and branch pricing described above can be performed. Alternatively, in other embodiments, processor 11 can be configured to perform the method for integrated scheduling of container terminal resources based on reinforcement learning and branch pricing by any other suitable means (e.g., by means of firmware).

[0154] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0155] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0156] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0157] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0158] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0159] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0160] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0161] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A container terminal resource integrated scheduling method based on reinforcement learning and branch pricing, characterized in that, include: Obtain scheduling correlation data for container terminal scheduling, and construct an original model describing the joint allocation problem of berths, quay cranes and yards based on the scheduling correlation data; wherein, the scheduling correlation data includes basic ship data, terminal resource data and cost rule data, and the original model aims to minimize the total operating cost and includes resource and time-related constraints; Based on the block structure of the original model, the original model is decomposed into a main problem and multiple pricing sub-problems. A pre-constructed branch pricing framework is used to solve the decomposed main problem and multiple pricing sub-problems to obtain linearly relaxed optimal solutions and integer feasible solutions. The main problem involves combining and selecting from multiple feasible single-ship complete scheduling plans to satisfy global resource constraints. Each pricing sub-problem is responsible for generating a new feasible scheduling plan with the lowest cost for a single ship. The branching process of the aforementioned branch pricing framework is modeled as a tree Markov decision process. A three-part graph representing the decision state is constructed based on the link relationships between branch variables, main problem column nodes, and main problem constraint nodes. Based on the three-part graph and the reinforcement learning dynamic branching strategy, the target scheduling scheme is determined, including: The optimal branch decision of the model output is obtained by inputting the three-part graph into a pre-trained graph convolutional neural network. The optimal branch decision is embedded into the branch pricing framework, and a target scheduling scheme that satisfies global optimality is determined through branch bounding and pruning mechanisms. The step of inputting the three-part graph into a pre-trained graph convolutional neural network to obtain the optimal branch decision of the model output includes: The three-part graph is input into a pre-trained graph convolutional neural network. The graph convolutional layer aggregates the association information between the branch variables, the main problem column nodes and the main problem constraint nodes, and outputs the selection probability of each candidate branch variable. Based on multiple selection probabilities, the candidate branch variable with the highest selection probability is selected as the optimal branch decision; The three-part graph representing the decision state, constructed based on the linking relationships between branch variables, main problem column nodes, and main problem constraint nodes, includes: Determine the candidate branch variable nodes, the main problem column nodes, and the main problem constraint nodes; Add features to the candidate branch variable node, the main problem column node, and the main problem constraint node respectively; Links are constructed between nodes after adding features to obtain a tripartite graph representing the decision state.

2. The method according to claim 1, characterized in that, The pre-constructed branch pricing framework is used to solve the decomposed main problem and multiple pricing subproblems to obtain linearly relaxed optimal solutions and integer feasible solutions, including: The decomposed main problem and pricing subproblems are solved iteratively based on a pre-built branch pricing framework; The optimal solution for linear relaxation is obtained through a column generation mechanism, and an integer feasible solution is generated by combining the master heuristic algorithm.

3. The method according to claim 2, characterized in that, Features are added to the candidate branch variable nodes, the main problem column nodes, and the main problem constraint nodes, respectively, including: Add variable solution values, corresponding ship and resource types, and historical branch counts to the candidate branch variable nodes; Add column cost, test number, and main variable values ​​to the main problem column nodes; Add dual optimal values ​​and features corresponding to the resource types of constraints to the main problem constraint nodes.

4. The method according to claim 1, characterized in that, The original model is a mixed integer programming model; the resource and time-related constraints include at least one of the following: The constraint that each vessel is allocated only one berth; Conflict constraints to prevent duplicate allocation of berths within the same time period; Capacity constraints that limit the total number of quay cranes in use; Time constraints that the vessel's operation time falls within the expected service time window; The vessel is designated to use the sub-block of its own company's yard, subject to ownership constraints.

5. The method according to claim 1, characterized in that, The target scheduling scheme includes the berth allocation results, quay crane configuration results, yard assignment results, and operation time arrangement results for each vessel.

6. A container terminal resource integration and scheduling system based on reinforcement learning and branch pricing, characterized in that, include: The original model building module is used to acquire scheduling correlation data of container terminal scheduling, and to build an original model describing the joint allocation problem of berths, quay cranes and yards based on the scheduling correlation data; wherein, the scheduling correlation data includes basic ship data, terminal resource data and cost rule data, and the original model aims to minimize the total operating cost and includes resource and time-related constraints; The model decomposition module is used to decompose the original model into a main problem and multiple pricing sub-problems based on the block structure of the original model. It solves the decomposed main problem and multiple pricing sub-problems based on a pre-built branch pricing framework to obtain linear relaxation optimal solutions and integer feasible solutions. The main problem involves combining and selecting from multiple feasible single-ship complete scheduling plans to meet global resource constraints. Each pricing sub-problem is responsible for generating a new feasible scheduling plan with the lowest cost for a single ship. The scheduling scheme determination module is used to model the branching process of the branch pricing framework as a tree Markov decision process, construct a three-part graph representing the decision state based on the link relationship between branch variables, main problem column nodes and main problem constraint nodes, and determine the target scheduling scheme based on the three-part graph and reinforcement learning dynamic branching strategy. The scheduling scheme determination module includes: The optimal branch decision acquisition unit is used to obtain the optimal branch decision output by the model by inputting the three-part graph into a pre-trained graph convolutional neural network. The scheduling scheme determination unit is used to embed the optimal branch decision into the branch pricing framework and determine the target scheduling scheme that satisfies global optimality through branch delimitation and pruning mechanisms. The optimal branch decision acquisition unit includes: The selection probability output subunit is used to input the tripartite graph into a pre-trained graph convolutional neural network, aggregate the association information between branch variables, the main problem column nodes and the main problem constraint nodes through the graph convolutional layer, and output the selection probability of each candidate branch variable; The optimal branch decision determination subunit is used to select the candidate branch variable with the highest selection probability as the optimal branch decision based on multiple selection probabilities. The scheduling scheme determination module includes: A node determination unit is used to determine candidate branch variable nodes, the main problem column nodes, and the main problem constraint nodes; The feature addition unit is used to add features to the candidate branch variable node, the main problem column node, and the main problem constraint node, respectively. The ternary graph construction unit is used to construct links between nodes after adding features, so as to obtain the ternary graph representing the decision state.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the container terminal resource integration scheduling method based on reinforcement learning and branch pricing as described in any one of claims 1-5.

Images