A container terminal automated scheduling method and system based on reinforcement learning

Abstract

The present invention discloses a container terminal automatic scheduling method and system based on reinforcement learning, the method comprising: obtaining port and ship information, initializing the environment according to the port and ship information, and constructing a container terminal environment to be simulated; in response to the current strategy scheduling instruction generated by the intelligent agent by inputting the container terminal environment into the pre-constructed strategy network, simulating the state of the future container terminal environment according to the Markov decision process to complete the berth and quay crane allocation; wherein the intelligent agent performs berth and quay crane scheduling according to the current strategy, updates the terminal state and checks whether there are ships to be scheduled, and if so, the intelligent agent generates new scheduling instructions according to environmental feedback and rewards until all ships complete the scheduling and unloading tasks, and completes the final berth and quay crane allocation. The present invention can reduce the total stay time of ships at container terminals and automatically complete the scheduling and unloading of ships.

Description

Technical Field

[0001] The present invention relates to a container terminal automated dispatching method and system based on reinforcement learning, belonging to the technical field of automated container terminals. Background Art

[0002] The core of the berth allocation and quay crane allocation optimization problem (BACAP) at container terminals is how to effectively reduce the total stay time of ships at the terminal when unloading, so as to shorten the delay of cargo delivery. In order to achieve this goal, it is necessary to reasonably allocate berths to ships arriving at the terminal and coordinate the quay crane resources at the terminal to improve the unloading efficiency.

[0003] Existing methods for solving BACAP problems can be divided into two categories: mathematical solvers and heuristic algorithms. Mathematical solvers express problems by constructing detailed mathematical models and use efficient solvers such as CPLEX to solve them. However, this method faces the problems of complex modeling and calculation processes and slow solution speed when dealing with large-scale problems, making it difficult to meet real-time requirements. In addition, when the constraints of the problem change, the model needs to be reconstructed, further reducing the computational efficiency.

[0004] Heuristic algorithms explore feasible solutions to problems based on specific rules. Compared with mathematical methods, heuristic algorithms are more popular because of their practicality and efficiency. These algorithms include dynamic programming, Monte Carlo tree search, and rolling horizon optimization. In recent years, population-based metaheuristic algorithms, especially evolutionary algorithms and swarm intelligence, have been widely studied and applied. These algorithms measure the utility of each solution in the population through evaluation functions, and guide the continuous evolution of solutions accordingly.

[0005] However, existing technologies have some structural defects in solving the BACAP problem. First, the limitation of computational efficiency makes mathematical solvers impractical in large-scale or rapidly changing operating environments. Second, existing algorithms are difficult to adapt to uncertain factors in maritime logistics, such as uncertainty in ship arrival times and operational failures of quay cranes, which may cause the initial solution to fail, increasing operational complexity and operating costs. Therefore, improvement measures need to optimize the computational framework, improve adaptability to changes in the actual operating environment, and reduce reliance on complex reconstruction or frequent adjustments to achieve a more economical and efficient berth and quay crane allocation optimization solution. Summary of the invention

[0006] The purpose of the present invention is to provide a container terminal automated scheduling method and system based on reinforcement learning, which can reduce the total stay time of ships when unloading at container terminals, shorten the delay time of cargo delivery, and automatically allocate ship berths and quay crane resources, thereby improving work efficiency.

[0007] In order to solve the above technical problems, the present invention is implemented by adopting the following technical solutions.

[0008] In a first aspect, the present invention provides a container terminal automated scheduling method based on reinforcement learning, comprising:

[0009] Acquire port and ship information, initialize the environment according to the port and ship information, and construct a container terminal environment to be simulated;

[0010] In response to the current policy scheduling instruction generated by the agent by inputting the container terminal environment into the pre-built policy network, the state of the future container terminal environment is simulated according to the Markov decision process to achieve berth and quay crane allocation;

[0011] Among them, the state of the future container terminal environment is simulated according to the Markov decision process to complete the berth and quay crane allocation, including:

[0012] After scheduling according to the current policy scheduling instructions, update the status of the container terminal environment to determine whether there are ships waiting for scheduling;

[0013] If there are ships waiting to be dispatched, the updated state of the container terminal environment and the corresponding rewards will be fed back to the agent, and a new strategic dispatch instruction will be generated by the agent;

[0014] After scheduling according to the new strategy scheduling instructions, the status of the container terminal environment is updated again, and it continues to be determined whether there are ships waiting for scheduling until all ships complete scheduling, unloading and leave the port, realizing the final berth and quay crane allocation.

[0015] In combination with the first aspect, further, the port and ship information includes quay cranes, berthing areas and ship schedules, wherein the ship schedule includes information on ships that are scheduled to enter the port for unloading within a predetermined time period.

[0016] In combination with the first aspect, further, the agent generates a current policy scheduling instruction by inputting the container terminal environment into a pre-built policy network, including:

[0017] Obtain the ship status of the container terminal environment; wherein the ship status includes the berthed ships , Waiting for the ship and arriving ships ;

[0018] The moored vessel After being sorted by priority, the information is input into the strategy network to generate a first state information vector and the waiting ship and arriving ships The arrival time of the ships is sequentially input into the strategy network to generate the second state information vector ;

[0019] The first state information vector and the second state information vector Input to their respective fully connected layers to obtain the first processed state information vector and the second processed state information vector , and the first processed state information vector and the second processed state information vector Merge to get the global state information vector ;

[0020] The global state information vector , Waiting for the ship and arriving ships Generate the first query vector through a single-layer fully connected network , the first key vector and the first value vector ;

[0021] Using the first query vector , the first key vector Calculate the attention weight and apply it to the first value vector Weighted, get the target ship information vector;

[0022] Calculate the berth area vector of the target ship according to the target ship information vector and the state of the available berth area in the container terminal environment, wherein the available berth area includes a location set of the berth area and a corresponding quay crane set;

[0023] The target ship information vector is input into a single-layer fully connected layer to generate a second query vector , and input the berth area vector of the target ship into two single-layer fully connected layer networks to generate the second key vector and the second value vector ;

[0024] Using the second query vector , the second key vector And calculate the attention weight, and the second value vector Weighted, get the ship arrangement vector;

[0025] The ship arrangement is input into a single-layer fully connected layer and normalized to obtain the target ship in continuous free berths and continuous idle quay cranes The ratio within the range, and the priority of the target ship ;

[0026] According to the target ship in the continuous free berth and continuous idle quay cranes Ratio within range and priority of target ships , calculate the final berth position, quay crane number and ship identification number to generate the final strategic scheduling instructions.

[0027] In combination with the first aspect, further, after scheduling according to the current strategy scheduling instruction, the state of the container terminal environment is updated to determine whether there is a ship waiting for scheduling. If there is a ship waiting for scheduling, the updated state of the container terminal environment and the corresponding reward are fed back to the intelligent agent, and a new strategy scheduling instruction is generated by the intelligent agent, specifically including:

[0028] Determine whether the strategic scheduling instruction is valid. If the strategic scheduling instruction is valid, perform scheduling according to the current strategic scheduling instruction and update the state of the container terminal environment to be simulated, and determine whether there is a ship waiting for scheduling;

[0029] If there are ships waiting for scheduling, the updated state of the container terminal environment and the corresponding rewards will be fed back to the intelligent agent, and new strategy scheduling instructions will be regenerated by the intelligent agent; if the intelligent agent outputs invalid strategy scheduling instructions, berths and quay cranes will no longer be allocated to the ships waiting in the current container terminal environment to be simulated.

[0030] In combination with the first aspect, further, the step of feeding back the updated state of the container terminal environment and the corresponding reward to the agent includes:

[0031] Continuously generate new policy scheduling instructions and update the status of the container terminal environment, and accumulate rewards at all times based on the updated status of the container terminal environment;

[0032] Feed the accumulated rewards at all times back to the density diversity enhanced evolution strategy optimizer in the agent;

[0033] The density diversity enhanced evolutionary strategy optimizer utilizes the accumulated rewards at all times to continuously adjust the parameter values ​​of the strategy network, so that the intelligent agent selects the optimal strategy scheduling action in the future container terminal environment.

[0034] In combination with the first aspect, further, the density diversity enhanced evolutionary strategy optimizer continuously adjusts the parameter value of the strategy network using the accumulated rewards at all times, including:

[0035] Constructing a set containing the historical distribution means of all ship berths and quay crane allocations, and initializing the parameters of the evolutionary strategy optimizer;

[0036] The sampling kernel density estimation method samples the policy network parameters based on the mean set of historical distributions of all ship berths and quay crane allocations;

[0037] Wherein, a density distribution map is constructed at each sampling, each policy network parameter in the density distribution map is evaluated, and it is determined whether each policy network parameter is located in a low-density area, and the policy network parameters located in the low-density area are taken as candidate solutions, and for each candidate solution, the policy network is executed and the accumulated reward of each candidate solution in the environment module is calculated;

[0038] Based on the accumulated rewards, calculate the ranking changes of all ship berths and quay crane allocations, and determine the new distribution means of all ship berths and quay crane allocations according to the new ranking changes;

[0039] Based on the new distribution mean and in combination with the accumulated reward value, the path, ranking difference, cumulative ranking ratio and mutation step in the evolution strategy optimizer are updated. If the cumulative ranking ratio and mutation step meet the restart requirements, the current strategy network parameters are reset until the expected number of resets is reached.

[0040] In combination with the first aspect, further, the expression of the density distribution map is:

[0041] (1);

[0042] in, represents the estimated value of the probability density function at point x; n represents the size of the sample data set; represents bandwidth; i represents the sample number of the ship; represents the kernel function; Indicates the point where density needs to be calculated; represents the sample of the i-th ship, and X represents the sample data set used for estimation.

[0043] In combination with the first aspect, further, continuing to determine whether there are ships waiting for scheduling until all ships complete scheduling, unload and leave the port includes:

[0044] If there are ships that continue to wait for scheduling, the remaining ships are continuously scheduled through the strategic scheduling instructions generated by the agent until all the ships waiting for scheduling are scheduled; if there are no ships waiting for scheduling, the remaining idle quay cranes are allocated according to the current strategic scheduling instructions and the priority of the moored ships; wherein the remaining idle quay crane allocation process includes:

[0045] The current strategy scheduling instructions are used to determine whether the anchored ships need to seize the quay cranes. If the number of quay cranes allocated by the agent to each ship is less than the preset maximum number of quay cranes, the quay cranes are seized according to the priority of the ship; if the number of quay cranes allocated by the agent to the ship is equal to the preset maximum number of quay cranes, the ship does not seize the quay cranes.

[0046] In combination with the first aspect, further, the constraint conditions for achieving the final allocation of ships and quay cranes include:

[0047] (i) Maintaining safe berthing distances between vessels when their stays overlap:

[0048] (2);

[0049] in, represents the berth position of any ship; s represents the safe berth distance ratio when the ship is berthed; Indicates the length of any ship; Indicates the length of the adjacent ship; Indicates the berth position of adjacent ships;

[0050] (ii) Maximum number of quay cranes:

[0051] (3);

[0052] Where m represents the total number of quay cranes; j represents the index of the quay crane; represents the unloading time of the jth quay crane for the i-th ship; represents the i-th ship; Represents a collection of ships; represents the indicator function; represents the maximum allowed number of quay cranes for the i-th ship;

[0053] (iii) The quay crane can only start unloading the cargo of another ship after completing the unloading of the cargo of one ship:

[0054] (4);

[0055] in, Indicates the start time of unloading of the i-th ship by the j-th quay crane; represents the time when the i-th ship leaves the container terminal;

[0056] (iv) The operation sequence of the quay crane should be consistent with the berthing sequence of the ship:

[0057] (5);

[0058] in, Indicates The number of the quay crane used to unload the vessel; Indicates The number of the quay crane used to unload the vessel;

[0059] (v) A quay crane can only serve one ship at a time:

[0060] (6);

[0061] in, Indicates the jth quay crane to the The time when the unloading of the vessel starts; Indicates the jth quay crane to the duration of unloading of the vessel; Indicates the jth quay crane to the The time when the unloading of the vessel starts; Indicates the jth quay crane to the The duration of unloading of the vessel.

[0062] In a second aspect, the present invention provides a container terminal automated dispatching system based on reinforcement learning, comprising an environment module and an intelligent agent;

[0063] The environment module is used to obtain port and ship information, initialize the environment according to the port and ship information, and construct a container terminal environment to be simulated;

[0064] The environment module is also used to respond to the current policy scheduling instruction generated by the intelligent agent by inputting the container terminal environment into the pre-built policy network, and simulate the state of the future container terminal environment according to the Markov decision process to complete the berth and quay crane allocation;

[0065] Among them, the state of the future container terminal environment is simulated according to the Markov decision process to complete the berth and quay crane allocation, including:

[0066] After the environment module schedules the berth and quay crane according to the current strategy scheduling instructions, it updates the status of the container terminal environment and determines whether there are ships waiting for scheduling;

[0067] If there are ships waiting to be dispatched, the environment module will feed back the updated state of the container terminal environment and the corresponding rewards to the agent, and the agent will generate new strategic dispatch instructions based on the rewards;

[0068] After the environment module performs scheduling according to the new strategy scheduling instructions, it updates the status of the container terminal environment again and continues to determine whether there are ships waiting for scheduling until all ships complete scheduling, unloading and leave the port, completing the final berth and quay crane allocation.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] The core of the automated scheduling method of the present invention is to use the technology of reinforcement learning to enable the continuous interaction between the intelligent agent and the environment, thereby realizing the continuous optimization of the terminal operation process. The intelligent agent uses the pre-built strategy network to generate scheduling instructions based on real-time port and ship information to simulate and determine the best berth and quay crane allocation plan. With each scheduling decision, the intelligent agent will adjust its strategy based on the feedback provided by the environmental module, that is, the update of the terminal status and the corresponding rewards. This real-time feedback mechanism enables the intelligent agent to learn which decisions are effective and which ones need to be improved, thereby gradually improving its decision-making quality. In this way, the terminal can more flexibly respond to the uncertainty of ship arrival and sudden changes in the operation process, ensuring that resources are used most effectively. The continuous learning and adaptation of the intelligent agent enables the terminal to improve operating efficiency, reduce the waiting time of ships in the port, and speed up the turnover of goods while ensuring safety.

[0071] In addition, the agent in the reinforcement learning framework is able to adaptively adjust its strategy to cope with the ever-changing conditions and unexpected situations in the container terminal environment. This adaptability is a major advantage of reinforcement learning, which enables the agent to handle complex and dynamic scheduling challenges without human intervention. In this way, the agent is able to continuously improve the quality of its decisions, thereby improving the overall operational efficiency and adaptability of the terminal. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] Figure 1 A flowchart of a container terminal automation method based on reinforcement learning provided by an embodiment of the present invention;

[0073] Figure 2 A schematic diagram of an agent strategy network provided by an embodiment of the present invention;

[0074] Figure 3 A schematic diagram of an evolutionary strategy optimizer based on density diversity enhancement provided by an embodiment of the present invention;

[0075] Figure 4 A schematic diagram of evolutionary reinforcement learning based on density diversity enhancement provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0076] The technical solution of the present invention is described in detail below through the accompanying drawings and specific embodiments. It should be understood that the embodiments of the present invention and the specific features in the embodiments are detailed descriptions of the technical solution of the present invention, rather than limitations on the technical solution of the present invention. The embodiments of the present invention and the technical features in the embodiments may be combined with each other unless there is a conflict.

[0077] The term "and / or" is only a description of the association relationship between related objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " generally indicates that the related objects are in an "or" relationship.

[0078] Example 1

[0079] It should be noted that the agent is the decision maker, while the environment module is the background of the problem, that is, the container terminal. The agent and the environment module will interact many times throughout the process.

[0080] See also Figure 1 This embodiment introduces a container terminal automatic scheduling method based on reinforcement learning, including the following steps:

[0081] Step S1, obtaining port and ship information, initializing the environment according to the port and ship information, and constructing a container terminal environment to be simulated;

[0082] Step S2, in response to the current policy scheduling instruction generated by the agent by inputting the container terminal environment into the pre-built policy network, simulating the state of the future container terminal environment according to the Markov decision process to complete the berth and quay crane allocation;

[0083] Among them, the state of the future container terminal environment is simulated according to the Markov decision process to complete the berth and quay crane allocation, including:

[0084] After berth and quay crane scheduling is performed according to the current strategy scheduling instructions, the status of the container terminal environment is updated to determine whether there are ships waiting for scheduling;

[0085] If there are ships waiting to be dispatched, the updated state of the container terminal environment and the corresponding rewards will be fed back to the agent, and a new strategic dispatch instruction will be generated by the agent;

[0086] After scheduling according to the new strategy scheduling instructions, the status of the container terminal environment is updated again, and it continues to be determined whether there are ships waiting for scheduling until all ships have completed scheduling, unloading and leaving the port, completing the final berth and quay crane allocation.

[0087] The container terminal automation system uses reinforcement learning technology to continuously learn and optimize scheduling strategies through real-time interaction between the intelligent agent and the terminal operating environment. The intelligent agent automatically adjusts its decisions to cope with changing terminal conditions based on environmental feedback, such as ship waiting time and resource utilization. This intelligent strategy scheduling and environmental simulation not only improves operational efficiency, but also reduces operating costs. Reinforcement learning enables terminals to flexibly adapt to environmental changes, such as uncertainty in ship arrivals and fluctuations in cargo flows, ensuring that resource allocation plans maximize the utilization of port resources while minimizing the total stay time of ships. This process improves operational efficiency, ensures smooth port operations, and enhances economic efficiency. By reducing the need for manual scheduling, the system reduces operating costs and improves the overall competitiveness of ports, reflecting the important value of reinforcement learning in improving intelligent decision support.

[0088] Example 2

[0089] See also Figure 4 In order to solve the problem of allocating ship berths and quay cranes in container terminals, this paper adopts evolutionary reinforcement learning based on density diversity enhancement (ERL-DDE). This method achieves the goal of reasonably allocating quay cranes and berths through the interaction between the environment module and the intelligent agent, thereby significantly improving the terminal operation efficiency and resource utilization.

[0090] Step S1, obtaining port and ship information, initializing the environment according to the port and ship information, and constructing a container terminal environment to be simulated;

[0091] Here, the port and ship information includes quay cranes, berth areas, and ship schedules; among them, quay cranes include the number of quay cranes, quay crane unloading efficiency, etc., berth areas include ship berthing positions, ship berthing time, etc., and ship schedules include ship information that plans to enter the port to unload within a predetermined time period, such as ship length, loaded cargo volume, ship arrival time, etc. The specific port and ship information variables are as follows:

[0092] Environmental constants

[0093] Container terminal berth length: ;

[0094] Safe berth distance ratio when the ship is berthing: .

[0095] Ship constants

[0096] Total number of ships: ;

[0097] Ship Collection: ,in, Indicates the first ship; Indicates the nth ship.

[0098] Ship length collection: ,in, Indicates the length of the first ship; Indicates the length of the nth ship.

[0099] Ship's initial cargo load collection: ,in, Indicates the cargo loading of the first ship; Indicates the cargo loading capacity of the nth ship.

[0100] Ship arrival time collection: ,in, Indicates the arrival time of the first ship; Indicates the arrival time of the nth ship.

[0101] Maximum number of ship unloading quay cranes: ,in, Indicates the maximum number of quay cranes for unloading the first ship; Indicates the maximum number of quay cranes for unloading the nth ship.

[0102] Quay crane constant

[0103] Total number of quay cranes: .

[0104] Quay crane assembly: ,in, Indicates the first assigned quay crane; The mth assigned quay crane.

[0105] Efficiency matrix of quay crane unloading: ,in, It indicates the unloading efficiency of the first quay crane for the first ship; It represents the unloading efficiency of the mth quay crane for the first ship; It represents the unloading efficiency of the first quay crane for the nth ship; It represents the unloading efficiency of the mth quay crane for the nth ship.

[0106] Ship decision variables

[0107] Ship berthing location collection: ,in, Indicates the berthing position of the first vessel; Indicates the berthing position of the nth ship.

[0108] Vessel berthing time collection: ,in, Indicates the first ship's berthing time; Indicates the berthing time of the nth ship.

[0109] Vessel departure time collection: ,in, Indicates the time when the first ship leaves the container terminal; Indicates the time when the nth ship leaves the container terminal.

[0110] Vessel Priority Sets: ,in, Indicates the first ship priority; Indicates the priority of the nth ship.

[0111] Quay Crane Decision Variables

[0112] Quay crane unloading start time matrix: ,in, Indicates the start time of unloading the first ship by the first quay crane; Represents the start time of unloading the first ship by the mth quay crane; Represents the start time of unloading of the mth ship by the first quay crane; Represents the start time of unloading of the nth ship by the mth quay crane.

[0113] Quay crane unloading duration matrix ,in, Represents the duration of unloading of the first ship by the first quay crane; Represents the duration of unloading the first ship by the mth quay crane; Represents the duration of unloading of the mth ship by the first quay crane; Represents the duration of unloading the nth ship by the mth quay crane.

[0114] Step S2, in response to the current policy scheduling instruction generated by the agent by inputting the container terminal environment into the pre-built policy network, simulating the state of the future container terminal environment according to the Markov decision process to complete the berth and quay crane allocation;

[0115] Among them, the state of the future container terminal environment is simulated according to the Markov decision process to complete the berth and quay crane allocation, including:

[0116] After berth and quay crane scheduling is performed according to the current strategy scheduling instructions, the status of the container terminal environment is updated to determine whether there are ships waiting for scheduling;

[0117] If there are ships waiting to be dispatched, the updated state of the container terminal environment and the corresponding rewards will be fed back to the agent, and a new strategic dispatch instruction will be generated by the agent;

[0118] After scheduling according to the new strategy scheduling instructions, the status of the container terminal environment is updated again, and it continues to be determined whether there are ships waiting for scheduling until all ships have completed scheduling, unloading and leaving the port, completing the final berth and quay crane allocation.

[0119] Specifically, the objects of the agent strategy scheduling instructions are ships and quay cranes, with the purpose of optimizing the berth allocation of ships and the allocation of quay cranes. In order to reduce the scale of the strategy network, the present invention integrates quay crane scheduling into the ship priority allocation, and the decision of the agent is limited to the waiting ships, and determines the target ship for scheduling, the berth position of the target ship, and the initial quay crane allocation of the target ship among the waiting ships.

[0120] Among them, the decision-making tasks include two discrete selection tasks and three continuous numerical tasks. The discrete selection tasks include deciding which ship to select as the target ship and which idle berth area to select for berthing; the numerical tasks include determining the precise berthing position of the ship in the berthing area, the specific serial number of the unloading quay crane assigned to the target ship, and the priority of the ship.

[0121] See also Figure 2 Step S21: The agent receives status information from the container terminal environment, including the status of the moored ships. , Waiting for the ship and arriving ships , the status information of the ship is expressed as:

[0122]

[0123] in, Indicates the i-th ship that has anchored; Indicates that the i-th ship is waiting; represents the i-th incoming ship; represents the time step of waiting for the ship; represents the berthing position of the i-th ship; represents the length i of the i-th ship; Represents the index of the first quay crane occupied by the i-th ship at time t; represents the total number of quay cranes occupied by the i-th ship at time t; Indicates the start waiting time for the waiting ship; represents the cargo loading capacity of the i-th ship at time t; represents the end waiting time of waiting for the ship; t represents the current moment or decision moment.

[0124] Step S22: The moored vessel After sorting by priority, it is input into the GRU network to generate the first state information vector ; will be waiting for the ship and arriving ships The data are input into the GRU network in the order of the arrival time of the ships to generate the second state information vector = ;

[0125] It should be noted that the GRU network can record important ship information and forget unimportant information. These important information together constitute the global state information of the entire environment and are used for the selection of ship berths.

[0126] Step S23: The first state information vector and the second state information vector Input to their respective fully connected layers to obtain the first processed state information vector and the second processed state information vector , and the first processed state information vector and the second processed state information vector Merge to get the global state information vector , the global state information vector The expression is:

[0127] (2)

[0128] (3)

[0129] in, A state information vector representing a ship that has been moored; A state information vector representing a ship that has not yet moored; Indicates the state information vector of the moored ship; Indicates the state information vector of the ship that has not yet moored; The bias term of the neural network that processes the state of the moored ship; The bias term of the neural network that processes the state of the moored ship; A weight matrix representing a neural network for processing the state of a vessel that has not yet docked; Represents the weight matrix of the neural network that handles the state of a ship that has not yet docked.

[0130] Since the decision made includes discrete actions and continuous actions, and the effective action space of discrete actions often changes, the above action of outputting global state information through the fully connected layer cannot meet the requirements. In order to ensure that the strategic scheduling action instructions generated by the agent meet the constraints of ship berth and quay crane allocation, the Attention mechanism is introduced to ensure that the strategic scheduling action meets the constraints. The expression of the Attention mechanism is:

[0131] (4)

[0132] Among them, Q represents the query matrix, including the type and characteristics of the environmental status information to be queried; K represents the key matrix, which is multiplied with the query matrix Q to measure the degree of matching to determine whether the input data needs to be focused on; V represents the value matrix, which contains the specific content of the current task, and the value matrix is ​​weighted to obtain the final output; represents the key matrix transpose; Represents the query vector dimension.

[0133] Step S24: Transform the global state information vector and waiting for the ship and arriving ships Generate the first query vector through a single-layer fully connected network , the first key vector and the first value vector ;

[0134] Generate the first query vector , the first key vector and the first value vector The expression is:

[0135] (5)

[0136] in, represents the ship query weight matrix; represents the ship bond weight matrix; represents the ship value weight matrix; Represents the global state information vector; Indicates the state vector of the ship that has not yet moored; Indicates the ship query bias item; represents the ship key offset term; represents the ship value bias term, represents the ship query matrix; represents the ship bond matrix; Represents the ship value matrix.

[0137] Step S25: Using the first query vector , the first key vector Calculate the attention weight and add the first value vector Weighted, get the target ship information vector ; The target ship information vector The expression is:

[0138] (6)

[0139] in, Represents the target ship information vector.

[0140] It should be noted that the attention weight represents the preference of the value vector. The higher the weight, the more attention is paid to a certain part of the target ship, and the strategy scheduling action instruction selects the ship with the highest weight. The expression for selecting the ship with the highest weight is:

[0141]

[0142] in, Indicates the ship with the highest weight; Indicates the ship serial number; Indicates the total number of ships.

[0143] Step S26, calculating the berth area vector of the target ship according to the target ship information vector and the state of the available berth area;

[0144] Specifically, the status of the continuous available berth area of ​​the target ship is calculated based on the length and position of the ship ; Wherein, the available berth area includes the location set of the available berth area And the corresponding gantry crane assembly .

[0145] Merge the location set of the available berth area and the corresponding quay crane set to obtain the status of the available berth area , the expression is:

[0146] (8)

[0147] in, Indicates the starting and ending points of the position segment for recording the berth area; It indicates the starting number and the ending number of the available quay cranes at the current berth area; Indicates the status of available berth areas.

[0148] As in the above formula (6), in order to ensure that the strategic scheduling action meets the constraint conditions, the embodiment of the present invention also selects the berth area where the target ship should berth through the attention mechanism. The specific selection process is as follows:

[0149] Step S211, calculating the berthing area vector of the target ship according to the target ship information vector and the state of the available berthing area, wherein the berthing area includes a position set of available berthing areas and a corresponding quay crane set;

[0150] Step S212: Input the target ship information vector into a single fully connected layer to generate a second query vector , and input the berth area vector of the target ship into two single-layer fully connected layer networks to generate the second key vector and the second value vector ;

[0151] Specifically, generate the second query vector , the second key vector and the second value vector The expression is:

[0152] (9)

[0153] in, represents the berth query weight matrix; represents the berth key weight matrix; represents the berth value weight matrix; Indicates the berth query bias item; Indicates the berth key offset item; Indicates the berth value bias item; represents the berth query matrix; represents the berth key matrix; Represents the berth value matrix.

[0154] Step S213: Using the second query vector , the second key vector Calculate the attention weight and add the second value vector Weighted, get the ship arrangement vector;

[0155] The expression of the ship arrangement vector is:

[0156] (10)

[0157] (11)

[0158] in, represents the parking information vector; Indicates the berth area where the vessel is arranged; Represents the berth-bond matrix transpose.

[0159] Step S214: Input the ship arrangement vector into a single-layer fully connected layer and perform normalization to obtain the target ship in continuous free berths. and continuous idle quay cranes The ratio within the range, and the priority of the target ship , whose expression is:

[0160] (12)

[0161] in, It indicates the proportion within the range of continuous free berths; Indicates the proportion within the range of continuous idle quay cranes; Indicates the target ship priority; represents the weight matrix; Represents the bias term.

[0162] Step S215: According to the target ship in the continuous idle berth and continuous idle quay cranes Ratio within range and priority of target ships , calculate the final berth position, quay crane number and ship identification number;

[0163] Step S216: Generate a final strategic scheduling instruction based on the calculated berth position, quay crane number and ship identification number.

[0164] The expression of the policy scheduling instruction is:

[0165] (13)

[0166] in, represents the docking position of the bow of the ship output by the agent; Indicates the end position of the selected free berth; Indicates the starting position of the selected free berth; It means the captain of a wooden ship; The sequence number of the initial quay crane required by the ship output by the intelligent agent is normalized; Indicates the normalization of the sequence number of the last quay crane that can be arranged in the selected free berth; Indicates the normalized value of the first quay crane that can be arranged in the selected free berth; Indicates the maximum number of quay cranes corresponding to the selected ship; Represents the normalized value of the number of quay cranes required for the target ship output by the agent; Indicates the maximum number of quay cranes for the selected ship; Represents the target ship priority output by the agent.

[0167] Furthermore, after the berth and quay crane are scheduled according to the current strategic scheduling instructions, the state of the container terminal environment is updated to determine whether there are ships waiting for scheduling; if there are ships waiting for scheduling, the updated state of the container terminal environment and the corresponding rewards are fed back to the intelligent agent, and a new strategic scheduling instruction is generated by the intelligent agent, specifically including:

[0168] Determine whether the strategic scheduling instruction is valid. If the strategic scheduling instruction is valid, perform scheduling according to the current strategic scheduling instruction and update the state of the container terminal environment to be simulated to determine whether there are ships waiting for scheduling;

[0169] If there are ships waiting for scheduling, the updated state of the container terminal environment and the corresponding rewards will be fed back to the intelligent agent, and new strategy scheduling instructions will be regenerated through the intelligent agent; if the intelligent agent outputs invalid strategy scheduling instructions, berths and quay cranes will no longer be allocated to ships waiting for scheduling in the current container terminal environment to be simulated.

[0170] It should be noted that the Markov decision process runs through the entire scheduling process and uses discrete events to effectively simulate the environment. The simulated state information is expressed by using the five-tuple Indicates that is the state space, is the action space, is the conversion function, is the reward after taking the action, is the discount factor. The embodiment of the present invention sets the discount factor is 1.

[0171] (I) State space

[0172] In order to simplify the problem, the state of the quay crane is integrated into the state of the ship, and then it is judged whether the ship has been fully unloaded and left the port. To limit the range of ships and quay bridges observed by the agent, so as to determine the Within the scope, whether the vessel has left the port.

[0173] Specifically, the embodiment of the present invention limits the sum of the moored ships and the waiting ships to never exceed If the total number of berthed vessels and waiting vessels is less than , then the global state will be supplemented by the upcoming ship state. In order to facilitate the agent to make wise decisions about the current state of the container terminal, the field of view All anchored ships and at least one waiting ship must be included, with the following constraints:

[0174] (14)

[0175] Among them, I( ) represents an indicator function; represents the field of view; i represents a ship; represents the berthing time of the i-th ship; represents the time when the i-th ship leaves the container terminal; represents the waiting time of the i-th propagation.

[0176] It should be noted that for each ship , the agent needs to track the following information to determine whether the ship has left the port:

[0177] Berth location ;

[0178] length ;

[0179] Cargo loading capacity ;

[0180] Index of the first occupied gantry crane ;

[0181] Total number of occupied quay cranes ;

[0182] Maximum number of quay cranes ;

[0183] Ship's stay time after arrival .

[0184] (II) Action Space

[0185] The remaining ships are scheduled according to the above new strategic scheduling instructions until all the ships waiting for scheduling are completed or the agent no longer generates new strategic scheduling instructions.

[0186] Specifically, since multiple ships may need to be scheduled at one time point, the embodiment of the present invention allows the agent to interact with the environment multiple times within one time step and execute multiple strategy scheduling instructions until all scheduling is completed. The expression of the action sequence that the agent can execute within one time step is:

[0187] (15)

[0188] in, represents the action sequence at time t; represents the action performed on the first ship at time t; represents the action performed on the second ship at time t.

[0189] Furthermore, the priority array is used to determine the execution order of the action set in formula (15), that is, to determine the priority of which ships are served by the quay crane. Therefore, the agent indirectly controls the quay crane decision variables (the start time of unloading the i-th ship by the j-th quay crane) through the priority management of the ship. and the duration of unloading of the jth quay crane on the i-th ship ), so that the agent does not need to directly handle the allocation of each quay crane, thereby reducing the complexity of strategy scheduling.

[0190] It should be noted that the agent selects the waiting ship from the ship status and determines the ship berth position, the sequence number of the first occupied quay bridge, the total number of occupied quay bridges and the ship priority, thereby generating a new strategy scheduling instruction. The expression of the new strategy scheduling instruction is:

[0191] (16)

[0192] in, Indicates the specific location where the vessel will be moored; Indicates the serial number of the first occupied quay crane; Indicates the total number of occupied quay cranes; Vessel priority; Indicates a new policy scheduling instruction.

[0193] (III) Rewards

[0194] Based on the current waiting ship, the environment module feeds back the reward of the previous time step of the updated state of the container terminal environment to the agent, and continuously accumulates the rewards. The evolutionary strategy optimizer based on density diversity enhancement uses the accumulated rewards to continuously adjust the parameter values ​​of the strategy network until each ship can complete unloading and leave the port within the optimal time.

[0195] It should be noted that the ultimate goal of reinforcement learning is to minimize the total stay time of all ships, that is, to make decisions through strategic scheduling instructions, thereby reducing the waiting and processing time of ships at container terminals, and the evolutionary strategy optimizer based on density diversity enhancement is learned based on the total reward value of the ship and quay crane actions, and does not require rewards for each step of the action, that is, the total reward value after the interaction between the environment module and the agent is set to the difference in stay time. The expression of the difference (reward) in stay time from one time step to another is:

[0196] (17)

[0197] in, represents the time step of the future environmental state; Indicates the time step of the current environment state; Indicates the time difference between the next key time node and the current moment; represents the arrival time of the i-th ship; i represents the ship sample number; n represents the total number of ships; Indicates whether the i-th ship has arrived at the port; represents the berthing time of the i-th ship; Indicates the time when the i-th ship leaves the container terminal.

[0198] (IV) Transfer function

[0199] In view of the continuity of time, the state cannot be transferred within a specific time step, but between discrete key decision time nodes. When the environment module receives the strategy scheduling instruction of the agent, the environment module verifies the legality of the strategy scheduling instruction. If the strategy scheduling action is legal, it is scheduled according to the strategy scheduling instruction and determines the decision variables of each ship. , , , , and , the expression of the decision scheduling action set is:

[0200] (18)

[0201] in, Indicates the index of the first quay crane occupied by the ship, Indicates the total number of occupied quay cranes, Indicates the berth position of the ship, Indicates the priority of the ship. Indicates the ship's unloading start time; Indicates the time when the quay crane starts unloading the ship; Indicates the index of the first quay crane occupied by the ship after the update. Indicates the total number of occupied quay cranes after update. Indicates the updated berth position of the ship, Indicates the priority of the updated ship. Indicates the current time.

[0202] When the environment module receives the dispatch action of formula (18), it checks whether there are any remaining ships among the waiting ships. If there are any remaining ships, the environment module continues to update the current state to the agent and continues to interact until there are no waiting ships. The state transfer function is used to transfer the state of the environment where there are currently no ships waiting, and enter the next key decision time point. The key decision time point is activated by two key events: a ship arrives at the container terminal and any ship completes unloading. The positioning between them is:

[0203] (19)

[0204] in, It indicates the time interval between the ship's arrival at the container terminal and the completion of unloading; Indicates the time interval between ships arriving at the container terminal; Indicates the time interval for a vessel to complete unloading.

[0205] Furthermore, the expression for the time interval between ship arrivals is:

[0206] (20)

[0207] in, represents the arrival time of the i-th ship; t represents the current time; Represents the set of times at which ships arrive.

[0208] The expression for the time interval for a ship to complete unloading is:

[0209] (twenty one)

[0210] in, Indicates the quay crane serial number; represents the remaining cargo volume of the i-th ship at time t; It represents the unloading volume per unit time of the i-th ship unloaded by the j-th quay crane.

[0211] if , it means that a ship has completed unloading and can leave the container terminal when the state is transferred, and the decision variables of the corresponding ship are updated, such as the departure time and the state of the quay crane after unloading.

[0212] (twenty two)

[0213] in, represents the departure time of the i-th ship; It represents the unloading time of the j-th quay crane for the i-th ship.

[0214] Furthermore, after the state transfer, the remaining cargo volume of the current ship is calculated, and its expression is:

[0215] (twenty three)

[0216] in, express The remaining cargo volume of the i-th ship at the time; Indicates the final serial number of the quay crane for which the i-th ship unloads; It represents the unloading volume per unit time of the i-th ship unloaded by the j-th quay crane.

[0217] According to the remaining cargo volume, the agent makes a decision. The environment module uses the state transfer function to change the state of the current container terminal environment. After the state is updated, the environment allocates the quay cranes. The specific quay crane allocation process includes:

[0218] Step S81: After scheduling according to the current strategic scheduling instruction, update the state of the container terminal environment, determine whether there are ships waiting for scheduling, and if there are ships waiting for scheduling, feed back the updated state of the container terminal environment and the corresponding rewards to the agent, and generate a new strategic scheduling instruction through the agent, which specifically includes:

[0219] Step S82: determine whether the strategic scheduling instruction is valid. If the strategic scheduling instruction is valid, perform scheduling according to the current strategic scheduling instruction and update the state of the container terminal environment to be simulated, and determine whether there is a ship waiting for scheduling;

[0220] Step S83: If there are ships waiting for scheduling, the updated state of the container terminal environment and the corresponding rewards will be fed back to the intelligent agent, and new strategy scheduling instructions will be regenerated through the intelligent agent; if the intelligent agent outputs invalid strategy scheduling instructions, berths and quay cranes will no longer be allocated to the ships waiting in the current container terminal environment to be simulated.

[0221] Specifically, during the allocation of quay cranes, the agent can query the ships that still have free space for quay cranes to unload after the first allocation, and determine whether there are idle quay cranes at both ends of the target ship. If there are idle quay cranes, the environment module allocates these quay cranes according to the priority of the ship and determines the start time of the ship's unloading. , and update the index of the first gantry crane , so that quay crane allocation occurs at each critical time node to ensure that the quay crane is optimally utilized, while the priority sorting of ships can reduce the time wasted by the agent when it first attempts to allocate quay cranes.

[0222] Finally, the environment module checks whether there are any ships waiting for the agent to dispatch. If there are no waiting ships, it enters the next key time decision point. At the same time, the environment module also checks whether all ships have completed unloading. If all ships have completed unloading, it indicates that the interaction between the agent and the environment module is completed, and the environment returns the accumulated rewards to the evolutionary strategy optimizer based on density diversity enhancement.

[0223] The environment returns the accumulated rewards to the density-diversity-enhanced evolution strategy optimizer, including:

[0224] Continuously generate new policy scheduling instructions and update the status of the container terminal environment, and accumulate rewards at all times based on the updated status of the container terminal environment;

[0225] Feed the accumulated rewards at all times back to the density diversity enhanced evolution strategy optimizer in the agent;

[0226] The density-diversity-enhanced evolutionary strategy optimizer uses the accumulated rewards at all times to continuously adjust the parameter values ​​of the strategy network, so that the agent can choose the optimal strategy scheduling action in the future container terminal environment.

[0227] Furthermore, the constraints for the final allocation of ships and quay cranes include:

[0228] (i) Maintaining safe berthing distances between vessels when their stays overlap:

[0229] (twenty four);

[0230] in, represents the berth position of any ship; s represents the safe berth distance ratio when the ship is berthed; Indicates the length of any ship; Indicates the length of the adjacent ship; Indicates the berth position of adjacent ships;

[0231] (ii) Maximum number of quay cranes:

[0232] (25);

[0233] Where m represents the total number of quay cranes; j represents the index of the quay crane; represents the unloading time of the jth quay crane for the i-th ship; represents the i-th ship; Represents a collection of ships; represents the indicator function; represents the maximum allowed number of quay cranes for the i-th ship;

[0234] (iii) The quay crane can only start unloading the cargo of another ship after completing the unloading of the cargo of one ship:

[0235] (26);

[0236] in, Indicates the start time of unloading of the i-th ship by the j-th quay crane; represents the time when the i-th ship leaves the container terminal;

[0237] (iv) The operation sequence of the quay crane should be consistent with the berthing sequence of the ship:

[0238] (27);

[0239] in, Indicates The number of the quay crane used to unload the vessel; Indicates The number of the quay crane used to unload the vessel;

[0240] (v) A quay crane can only serve one ship at a time:

[0241] (28);

[0242] in, Indicates the jth quay crane to the The time when the unloading of the vessel starts; Indicates the jth quay crane to the duration of unloading of the vessel; Indicates the jth quay crane to the The time when the unloading of the vessel starts; Indicates the jth quay crane to the The duration of unloading of the vessel.

[0243] Example 3

[0244] Example 3 is a further improvement on Example 2. In Example 2, a reinforcement learning-based container terminal automated scheduling method is introduced. The method optimizes the berth allocation of ships and the utilization of quay crane resources through the interaction between the agent and the environment module. However, Example 2 may not fully consider the "deceptive reward" problem that may be encountered in reinforcement learning, that is, the agent's strategy may converge to a local rather than a global optimal solution prematurely due to misleading rewards. In order to solve this problem, Example 3 introduces a niche strategy to enhance the behavioral diversity of individual ships and improve the overall search efficiency.

[0245] Specifically, several ship and quay crane allocation schemes are constructed, each of which consists of a group of individuals representing different parameter configurations of the policy network. Each ship and quay crane allocation scheme is independently optimized by a customized evolutionary strategy (ES) optimizer. However, in the optimization process of multiple sub-populations, evolutionary path overlap may occur, resulting in a reduction in global search diversity, which in turn affects resource utilization efficiency. The present invention proposes an innovative density-based diversity enhancement mechanism (DDE mechanism), which dynamically guides sub-populations to explore new areas by drawing a population exploration density distribution map to avoid them wandering in areas that have been over-searched. This method not only improves the global diversity of the search, but also helps the intelligent agent to discover better solutions, thereby making more accurate and effective strategy adjustments in complex and changing environments.

[0246] See also Figure 3 The DDE mechanism draws a population exploration density distribution map by utilizing the cumulative historical paths of all ship and quay crane allocation schemes. The explored areas are presented in dark colors (high density), while the unexplored areas are marked in light colors (low density). The purpose is to guide the ship and quay crane allocation schemes to avoid wandering in over-searched areas and explore new areas. In each round of iterative sampling steps, the DDE mechanism strategically avoids sampling points in those over-searched areas and encourages sub-populations to explore new areas.

[0247] Among them, the optimization process of the evolutionary strategy optimizer based on density diversity enhancement includes:

[0248] Step S51, construct a set containing the historical distribution means of all ship berths and quay crane allocations, and initialize the parameters of the evolutionary strategy optimizer;

[0249] It should be noted that before starting the optimization, the optimizer parameters are initialized, including the reorganization weights. , the number of elite individuals , Variance Efficient Selection Quality Etc., where the number of elite individuals determines the number of individuals participating in gradient calculation; the variance effective selection quality is used for normalization to keep the evolutionary path conforming to a Gaussian distribution with specific parameters.

[0250] Step S52: sampling kernel density estimation method, sampling the policy network parameters based on the historical distribution mean set of all ship berths and quay crane allocations;

[0251] Among them, a density distribution map is constructed at each sampling, each policy network parameter in the density distribution map is evaluated, and it is determined whether each policy network parameter is located in a low-density area, and the policy network parameters located in the low-density area are taken as candidate solutions. For each candidate solution, the policy network is executed and the accumulated reward of each candidate solution in the environment module is calculated;

[0252] Specifically, randomly generate sampling points, as The initial value of the scheduling scheme search distribution is used, and a set of candidate solutions is generated for each subpopulation in the Tth generation. The expression of the candidate solution is:

[0253] (29)

[0254] in, represents the Jth individual in the Ith strategy network parameter population; Represents the distribution mean of the I-th strategy network parameter population at the T-th iteration; Represents the mutation step size of the Tth iteration of the Ith strategy network parameter population; represents the rate of change of the covariance matrix; represents a random matrix that follows a standard normal distribution and is used to generate the Jth individual of the Ith strategy network parameter population; represents another random matrix that follows a standard normal distribution and is used to generate the Jth individual of the Ith strategy network parameter population; Represents the main search direction of the I-th strategy network parameter population at the T-th iteration.

[0255] Furthermore, to prevent repeated exploration, ES-DDE dispatches sampling points to high-density areas and guides the sampling process to avoid fully explored areas by constructing a density distribution map corresponding to the reward value.

[0256] The expression of the density distribution map is:

[0257] (30)

[0258] in, Represents the estimated value of the probability density function at point x; N represents the sample data set set; represents bandwidth; i represents the sample number of the ship; represents the kernel function; Indicates the point where density needs to be calculated; represents the sample of the i-th ship in X, where X represents the sample data set used for estimation.

[0259] Step S53: based on the accumulated rewards, calculate the ranking changes of all ship berths and quay crane allocations, and determine the new distribution mean of all ship berths and quay crane allocations according to the new ranking changes;

[0260] The expression of the updated historical distribution mean set is I:

[0261] (31)

[0262] in, represents the new distribution mean; represents the number of elite individuals; Indicates the serial number; represents the kth reorganization weight; Represents the kth individual in the Ith strategy network parameter population, sorted from high to low in terms of quality.

[0263] Step S54: Based on the new distribution mean and in combination with the accumulated reward value, the evolution path, ranking difference, cumulative ranking ratio and mutation step in the evolution strategy optimizer are updated. If the cumulative ranking ratio and mutation step meet the restart requirements, the current strategy network parameters are reset until the expected number of resets is reached.

[0264] Specifically, the updated evolutionary path expression is:

[0265] (32)

[0266] in, Indicates the main search direction of the I-th strategy network parameter population at the T+1-th iteration; Indicates the rate of change of the main search direction; Indicates the main search direction of the I-th strategy network parameter population at the T-th iteration; Represents the distribution mean of the I-th strategy network parameter population at the T-th iteration; Represents the mutation step size of the I-th strategy network parameter population at the T+1-th iteration; Denotes the variance effective selection quality.

[0267] The calculation expression for the sorting change is:

[0268] (33)

[0269] in, Table 1 Ranking changes of the population of strategy network parameters; Indicates that the total reward is higher than The number of individuals.

[0270] It should be noted that the cumulative ranking is used to adjust the search direction of the scheduling scheme, while the mutation step size determines the breadth of the population search. These two parameters work together to adjust the search direction and breadth of the population. The calculation expressions for the cumulative ranking and mutation step size are:

[0271] (34)

[0272] in, Represents the cumulative ranking rate of the I-th strategy network parameter population at the t+1-th iteration; represents the rate of change, which determines the number of iterations involved in the cumulative ranking rate; Represents the cumulative ranking rate of the I-th strategy network parameter population at the T-th iteration; Indicates the target ratio, the main factor that determines the change in cumulative ranking rate.

[0273] (35)

[0274] in, Represents the mutation step size of the I-th strategy network parameter population at the T+1-th iteration;

[0275] Represents the damping parameter that determines the amplitude of the mutation step size change.

[0276] Furthermore, if any policy network parameter fails to improve or meet the restart condition, the current policy network parameter is reset; the above process of generating new allocation decisions, evaluating and updating parameters is repeated until the optimal allocation solution is found.

[0277] Finally, as the number of iterations T increases, the number of elites Gradually increase to , where the number of iterations T is used to increase diversity in the early stage of the search, and to focus on searching for the optimal solution in the later stage. The calculation formula is as follows:

[0278] (36)

[0279] in, An iteration counter representing the ship and quay crane allocation scheme; Indicates the maximum number of elite individuals; represents the initial number of elite individuals; Indicates the number of elite individuals in the current iteration.

[0280] if becomes too large, parameters will be reset. In addition, if becomes too small or meets the restart condition of DDE, it will evaluate Whether the reward exceeds the average reward of the current iteration. If it exceeds, only the population parameters are reset; otherwise, the distribution mean point Both the population parameters will be reset to facilitate exploration of new search areas.

[0281] Example 4

[0282] An automated dispatching system for container terminals based on reinforcement learning, including an environment module and an intelligent agent;

[0283] The environment module is used to obtain port and ship information, initialize the environment according to the port and ship information, and build the container terminal environment to be simulated;

[0284] The environment module is also used to respond to the current policy scheduling instructions generated by the agent by inputting the container terminal environment into the pre-built policy network, and simulate the state of the future container terminal environment according to the Markov decision process to complete the berth and quay crane allocation;

[0285] Among them, the state of the future container terminal environment is simulated according to the Markov decision process to complete the berth and quay crane allocation, including:

[0286] After the environment module schedules the berth and quay crane according to the current strategy scheduling instructions, it updates the status of the container terminal environment and determines whether there are ships waiting for scheduling;

[0287] If there are ships waiting to be dispatched, the environment module will feed back the updated state of the container terminal environment and the corresponding rewards to the agent, and the agent will generate new strategic dispatch instructions based on the rewards;

[0288] After the environment module performs scheduling according to the new strategy scheduling instructions, it updates the status of the container terminal environment again and continues to determine whether there are ships waiting for scheduling until all ships complete scheduling, unloading and leave the port, completing the final berth and quay crane allocation.

[0289] The embodiments of the present invention are described above in conjunction with the accompanying drawings, but the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the enlightenment of the present invention, ordinary technicians in this field can also make many forms without departing from the scope of protection of the purpose of the present invention and the claims, which all fall within the protection of the present invention.

Claims

1. A container terminal automated scheduling method based on reinforcement learning, characterized in that: include: Acquire port and ship information, initialize the environment according to the port and ship information, and construct a container terminal environment to be simulated; In response to the current policy scheduling instruction generated by the agent by inputting the container terminal environment into the pre-built policy network, the state of the future container terminal environment is simulated according to the Markov decision process to achieve berth and quay crane allocation; The method of simulating the state of the future container terminal environment according to the Markov decision process to complete the allocation of berths and quay cranes includes: S1. After scheduling according to the current strategy scheduling instructions, update the status of the container terminal environment and determine whether there are ships waiting for scheduling; If there are ships waiting for scheduling, the updated state of the container terminal environment and the corresponding rewards are fed back to the intelligent agent, and a new strategy scheduling instruction is generated by the intelligent agent to determine whether the strategy scheduling instruction is valid. If the strategy scheduling instruction is valid, scheduling is performed according to the current strategy scheduling instruction and the state of the container terminal environment to be simulated is updated to determine whether there are ships waiting for scheduling; if there are ships waiting for scheduling, the updated state of the container terminal environment and the corresponding rewards are fed back to the intelligent agent, and a new strategy scheduling instruction is regenerated by the intelligent agent; if the intelligent agent outputs an invalid strategy scheduling instruction, berths and quay cranes are no longer allocated to the ships waiting for scheduling in the current container terminal environment to be simulated; Finally, the environment module checks whether there are any ships waiting for the agent to dispatch. If there are no ships waiting, it proceeds to the next key time decision point. At the same time, the environment module also checks whether all ships have completed unloading. If all ships have completed unloading, the environment returns the accumulated rewards to the evolutionary strategy optimizer based on density diversity enhancement. The density diversity enhanced evolutionary strategy optimizer continuously adjusts the parameter values ​​of the strategy network using the accumulated rewards at all times, including: Constructing a set containing the historical distribution means of all ship berths and quay crane allocations, and initializing the parameters of the evolutionary strategy optimizer; The sampling kernel density estimation method samples the policy network parameters based on the mean set of historical distributions of all ship berths and quay crane allocations; Wherein, a density distribution map is constructed at each sampling, each policy network parameter in the density distribution map is evaluated, and it is determined whether each policy network parameter is located in a low-density area, and the policy network parameters located in the low-density area are taken as candidate solutions, and for each candidate solution, the policy network is executed and the accumulated reward of each candidate solution in the environment module is calculated; Based on the accumulated rewards, calculate the ranking changes of all ship berths and quay crane allocations, and determine the new distribution means of all ship berths and quay crane allocations according to the new ranking changes; Based on the new distribution mean and in combination with the accumulated reward value, the path, ranking difference, cumulative ranking ratio and mutation step size in the evolution strategy optimizer are updated. If the cumulative ranking ratio and mutation step size meet the restart requirement, the current strategy network parameters are reset until the expected number of resets is reached; S2. After scheduling according to the new strategy scheduling instructions, the status of the container terminal environment is updated again, and it is continued to determine whether there are ships waiting for scheduling until all ships complete scheduling, unloading and leave the port, realizing the final berth and quay crane allocation.

2. The container terminal automated scheduling method based on reinforcement learning according to claim 1 is characterized in that: The port and vessel information includes quay cranes, available berthing areas and vessel schedules; The ship schedule includes information on ships that are scheduled to enter the port to unload cargo within a predetermined time period.

3. The container terminal automated scheduling method based on reinforcement learning according to claim 1, characterized in that: The agent generates a current policy scheduling instruction by inputting the container terminal environment into a pre-built policy network, including: Obtain the ship status of the container terminal environment; wherein the ship status includes the berthed ships , Waiting for the ship and arriving ships ; The moored vessel After being sorted by priority, the information is input into the strategy network to generate a first state information vector and the waiting ship and arriving ships The arrival time of the ships is sequentially input into the strategy network to generate the second state information vector ; The first state information vector and the second state information vector Input to each fully connected layer, and the state information vector processed by the fully connected layer and Merge to get the global state information vector ; The global state information vector , Waiting for the ship and arriving ships Generate the first query vector through a single-layer fully connected network , the first key vector and the first value vector ; Using the first query vector , the first key vector Calculate the attention weight and apply it to the first value vector Weighted, get the target ship information vector; Calculate the berth area vector of the target ship according to the target ship information vector and the state of the available berth area in the container terminal environment, wherein the available berth area includes a location set of the berth area and a corresponding quay crane set; The target ship information vector is input into a single-layer fully connected layer to generate a second query vector , and input the berth area vector of the target ship into two single-layer fully connected layer networks to generate the second key vector and the second value vector ; Using the second query vector , the second key vector And calculate the attention weight, and the second value vector Add the weights to obtain the ship arrangement vector; The ship arrangement is input into a single-layer fully connected layer and normalized to obtain the target ship in continuous free berths and continuous idle quay cranes The ratio within the range, and the priority of the target ship ; According to the target ship in the continuous free berth and continuous idle quay cranes The ratio of the range and the priority of the target ship , calculate the final berth position, quay crane number and ship identification number to generate the final strategic scheduling instructions.

4. The container terminal automated scheduling method based on reinforcement learning according to claim 1, characterized in that: The environment returns the accumulated rewards to the density-diversity-enhanced evolution strategy optimizer, including: Continuously generate new policy scheduling instructions and update the status of the container terminal environment, and accumulate rewards at all times based on the updated status of the container terminal environment; Feed the accumulated rewards at all times back to the density diversity enhanced evolution strategy optimizer in the agent; The density diversity enhanced evolutionary strategy optimizer utilizes the accumulated rewards at all times to continuously adjust the parameter values ​​of the strategy network, so that the intelligent agent selects the optimal strategy scheduling action in the future container terminal environment.

5. The container terminal automated dispatching method based on reinforcement learning according to claim 1, characterized in that: The expression of the density distribution map is: （1）； in, represents the estimated value of the probability density function at point x; n represents the size of the sample data set; represents bandwidth; i represents the sample number of the ship; represents the kernel function; Indicates the point where density needs to be calculated; represents the sample of the i-th ship, and X represents the sample data set used for estimation.

6. The container terminal automated dispatching method based on reinforcement learning according to claim 3 is characterized in that: The step of continuing to determine whether there are any ships waiting for dispatch until all ships have completed dispatch, unloaded and left the port includes: If there are ships that continue to wait for scheduling, the remaining ships are continuously scheduled through the strategic scheduling instructions generated by the agent until all the ships waiting for scheduling are scheduled; if there are no ships waiting for scheduling, the remaining idle quay cranes are allocated according to the current strategic scheduling instructions and the priority of the moored ships; wherein the remaining idle quay crane allocation process includes: The current strategy scheduling instructions are used to determine whether the anchored ships need to seize the quay cranes. If the number of quay cranes allocated by the agent to each ship is less than the preset maximum number of quay cranes, the quay cranes are seized according to the priority of the ship; if the number of quay cranes allocated by the agent to the ship is equal to the preset maximum number of quay cranes, the ship does not seize the quay cranes.

7. The container terminal automated dispatching method based on reinforcement learning according to claim 1, characterized in that: The constraints for achieving the final allocation of ships and quay cranes include: (i) Maintaining safe berthing distances between vessels when their stays overlap: （2）； in, represents the berth position of any ship; s represents the safe berth distance ratio when the ship is berthed; Indicates the length of any ship; Indicates the length of the adjacent ship; Indicates the berth position of adjacent ships; (ii) Maximum number of quay cranes: （3）； Where m represents the total number of quay cranes; j represents the index of the quay crane; represents the unloading time of the jth quay crane for the i-th ship; represents the i-th ship; Represents a collection of ships; represents the indicator function; represents the maximum allowed number of quay cranes for the i-th ship; (iii) The quay crane can only start unloading the cargo of another ship after completing the unloading of the cargo of one ship: （4）； in, Indicates the start time of unloading of the i-th ship by the j-th quay crane; represents the time when the i-th ship leaves the container terminal; (iv) The operation sequence of the quay crane should be consistent with the berthing sequence of the ship: （5）； in, Indicates The number of the quay crane used to unload the vessel; Indicates The number of the quay crane used to unload the vessel; (v) A quay crane can only serve one ship at a time: （6）； in, Indicates the jth quay crane to the The time when the unloading of the vessel starts; Indicates the jth quay crane to the duration of unloading of the vessel; Indicates the jth quay crane to the The time when the unloading of the vessel starts; Indicates the jth quay crane to the The duration of unloading of the vessel.

8. A container terminal automated dispatching system based on reinforcement learning, characterized in that: Includes environment module and agent; The environment module is used to obtain port and ship information, initialize the environment according to the port and ship information, and construct a container terminal environment to be simulated; The environment module is also used to respond to the current policy scheduling instruction generated by the intelligent agent by inputting the container terminal environment into the pre-built policy network, and simulate the state of the future container terminal environment according to the Markov decision process to complete the berth and quay crane allocation; Among them, the state of the future container terminal environment is simulated according to the Markov decision process to complete the berth and quay crane allocation, including: S1. After scheduling according to the current strategy scheduling instructions, update the status of the container terminal environment and determine whether there are ships waiting for scheduling; If there are ships waiting for scheduling, the updated state of the container terminal environment and the corresponding rewards are fed back to the intelligent agent, and a new strategy scheduling instruction is generated by the intelligent agent to determine whether the strategy scheduling instruction is valid. If the strategy scheduling instruction is valid, scheduling is performed according to the current strategy scheduling instruction and the state of the container terminal environment to be simulated is updated to determine whether there are ships waiting for scheduling; if there are ships waiting for scheduling, the updated state of the container terminal environment and the corresponding rewards are fed back to the intelligent agent, and a new strategy scheduling instruction is regenerated by the intelligent agent; if the intelligent agent outputs an invalid strategy scheduling instruction, berths and quay cranes are no longer allocated to the ships waiting for scheduling in the current container terminal environment to be simulated; Finally, the environment module checks whether there are any ships waiting for the agent to dispatch. If there are no ships waiting, it proceeds to the next key time decision point. At the same time, the environment module also checks whether all ships have completed unloading. If all ships have completed unloading, the environment returns the accumulated rewards to the evolutionary strategy optimizer based on density diversity enhancement. The density diversity enhanced evolutionary strategy optimizer continuously adjusts the parameter values ​​of the strategy network using the accumulated rewards at all times, including: Constructing a set containing the historical distribution means of all ship berths and quay crane allocations, and initializing the parameters of the evolutionary strategy optimizer; The sampling kernel density estimation method samples the policy network parameters based on the mean set of historical distributions of all ship berths and quay crane allocations; Wherein, a density distribution map is constructed at each sampling, each policy network parameter in the density distribution map is evaluated, and it is determined whether each policy network parameter is located in a low-density area, and the policy network parameters located in the low-density area are taken as candidate solutions, and for each candidate solution, the policy network is executed and the accumulated reward of each candidate solution in the environment module is calculated; Based on the accumulated rewards, calculate the ranking changes of all ship berths and quay crane allocations, and determine the new distribution means of all ship berths and quay crane allocations according to the new ranking changes; Based on the new distribution mean and in combination with the accumulated reward value, the path, ranking difference, cumulative ranking ratio and mutation step size in the evolution strategy optimizer are updated. If the cumulative ranking ratio and mutation step size meet the restart requirement, the current strategy network parameters are reset until the expected number of resets is reached; S2. After scheduling according to the new strategy scheduling instructions, the status of the container terminal environment is updated again, and it is continued to determine whether there are ships waiting for scheduling until all ships complete scheduling, unloading and leave the port, realizing the final berth and quay crane allocation.

Images