Container terminal truck scheduling method and device based on deep reinforcement learning

Abstract

The application discloses a container terminal truck scheduling method and device based on deep reinforcement learning, the method comprising: building a digital twin environment model corresponding to a real terminal, and completing the basic configuration of data, communication and parameters; according to the digital twin environment model, a unified state space for reinforcement learning intelligent agents is constructed; a dynamic action set is constructed for the current vehicle to be scheduled, and all candidate actions that violate the constraints are filtered out through the mask mechanism; a structure perception strategy network and a value network for truck scheduling are constructed, and advantage weighting update and multi-day rolling training are carried out; a composite reward is constructed to comprehensively measure vehicle efficiency, equipment utilization, empty mileage and task completion performance; a docking scheme including a data interface, an instruction issuing, a monitoring loop and a safety strategy is constructed to form a closed loop of training, deployment and operation. The application can overcome the problems in the prior art, such as the fragmentation of internal truck and external truck scheduling, the fragmentation of path planning, and the insufficient multi-objective optimization capability.

Description

Technical Field

[0001] This invention relates to a container terminal truck scheduling method and apparatus based on deep reinforcement learning, belonging to the field of container terminal scheduling and control. Background Technology

[0002] Container terminals are critical infrastructure for international trade and global supply chains, and their operational efficiency directly impacts ship turnaround time and the port's overall throughput capacity. Within the complex terminal system, sea-side quay cranes handle ship-to-shore loading and unloading, while land-side yard cranes handle yard operations. Horizontal transport is handled by internal trucks, while external trucks connect to the social transportation system via gates and yards. These elements are tightly coupled in both time and space; any local fluctuation in the operational rhythm can propagate upstream or downstream along the road network and queue links, leading to a chain reaction of equipment idleness, road congestion, and task backlog.

[0003] In traditional practice, truck scheduling has long relied on rules and experience, such as heuristic strategies like "first-come, first-served," "fixed priority," and "nearest allocation." These methods offer advantages like quick response and ease of implementation when tasks are small or subject to weak disturbances. However, in scenarios with dense shipping traffic, significant peaks and troughs in container volume, road network congestion, and frequent equipment status fluctuations, they often struggle to balance overall objectives: empty runs by vehicles are difficult to control, waiting times within the work area increase, and critical bottleneck equipment (especially quay cranes) experience intermittent idleness, leading to an imbalance in the entire operational chain. With the increasing automation and informatization of terminals, some mathematical programming methods and metaheuristic algorithms based on operations research have been introduced into the scheduling field. These methods can provide clearer goal and constraint modeling under static or semi-static conditions. However, under minute-level rolling optimization and large-scale concurrent tasks, they often face the dual challenges of computational complexity and uncertainty adaptability, making it difficult to reliably meet the timeliness requirements of online decision-making.

[0004] The complexity of actual production stems not only from the scale of the tasks but also from the diversity and overlap of constraints. On the vehicle side, there are load limits and safety distance constraints; on the equipment side, there are request limits and disabled / failed states; on the task side, there are pre-task operations and allocable time windows; and on the road network side, there are dynamic congestion and speed limits / road closures. Furthermore, it is necessary to maintain idempotent and traceable consistency with the Terminal Operations System (TOS). Coordination between internal and external trucks is also a major challenge: if only local optimization is performed around internal trucks, ignoring the reservation time windows and gate release rhythm of external trucks, "backflow" and "discontinuities" can easily form between the gates and the yard, causing deep congestion in the yard during peak hours. At the same time, the truck operation chain has a phased characteristic, typically including a leg towards the "target point" and a leg towards the "final point." If only the shortest route is allocated for a single task or a single stage, higher cross-regional handling costs can easily be incurred in subsequent stages, leading to a decrease in overall efficiency.

[0005] In recent years, deep reinforcement learning has attracted attention due to its end-to-end perception and policy self-learning capabilities. However, directly searching for policies in a large action space, without addressing action feasibility before policy normalization, frequently generates candidate actions that do not meet hard constraints such as load, queue, time window, or device state. This leads to training oscillations, extensive patching required for online deployment, and insufficient engineering usability. On the other hand, if the reward or objective function focuses only on a single dimension (e.g., only penalizing distance), the policy tends to oscillate between "local shortest distance" and "global link efficiency," resulting in persistent problems such as insufficient idle capacity on critical bottleneck devices and slow convergence at the tail end.

[0006] In summary, existing technologies still have the following four shortcomings: First, they lack "pre-decision feasible domain" control that is deeply coupled with the complex hard constraints of the port area. Constraints such as load, safety distance, and equipment status are often passively verified only after the action is generated, leading to frequent illegal actions, resulting in oscillations in strategy training and requiring numerous patches after deployment. Second, most existing methods handle one or two stages of the task separately, lacking a joint evaluation of cross-stage paths and cross-regional handling costs. This easily leads to locally optimal decisions between "nearest response" and "overall chain efficiency," resulting in subsequent empty runs and idle bottleneck equipment. Third, recent research and engineering deployments typically only model internal trucks, treating the reservation windows, gate release cycles, and yard interactions of external trucks as static inputs or even ignoring them entirely. There is a lack of unified state characterization and constraint transmission between internal and external trucks, making it difficult to avoid gate backflow and deep yard congestion during peak periods. Fourth, at the engineering implementation level, there is a lack of closed-loop design for TOS docking, instruction idempotency, clock consistency, and full-link auditing. Under minute-level rolling scheduling, it is difficult to guarantee the traceability and long-term stable operation of the strategy. Summary of the Invention

[0007] In view of this, the present invention provides a container terminal truck scheduling method, device, computer equipment and storage medium based on deep reinforcement learning. It can overcome the problems of fragmented scheduling of internal and external trucks, fragmented route planning and insufficient multi-objective optimization capabilities in the prior art. It performs unified modeling of task status, vehicle status (including internal and external trucks) and equipment status in a digital twin simulation environment, constructs a dynamic action space and introduces an action mask mechanism. Combined with multi-objective reward function and rolling window strategy learning, it realizes coordinated control of gate arrival time, yard crane / quay crane operation rhythm and road network congestion, thereby improving the overall operational efficiency and resource utilization of the terminal.

[0008] The first objective of this invention is to provide a container terminal truck scheduling method based on deep reinforcement learning.

[0009] The second objective of this invention is to provide a container terminal truck scheduling device based on deep reinforcement learning.

[0010] A third objective of this invention is to provide a computer device.

[0011] A fourth objective of this invention is to provide a computer-readable storage medium.

[0012] The first objective of this invention can be achieved by adopting the following technical solution:

[0013] A container terminal truck scheduling method based on deep reinforcement learning, the method comprising:

[0014] Build a digital twin environment model corresponding to the real port, and complete the basic configuration of data, communication and parameters;

[0015] Based on the digital twin environment model, a unified state space is constructed for use by reinforcement learning agents, which simultaneously covers the task set, vehicle set, and device set;

[0016] Based on a unified state space, a dynamic action set is constructed for the vehicle to be dispatched, and a masking mechanism is used to filter out all candidate actions that violate load, time window, process or equipment state constraints.

[0017] Based on a unified state space and action mask, a structure-aware policy network and a value network for truck scheduling are constructed, and advantage-weighted updates and multi-day rolling training are performed.

[0018] Based on a unified state space, a composite reward system is constructed to comprehensively measure vehicle efficiency, equipment utilization, empty mileage, and task completion performance.

[0019] Build a solution that integrates data interfaces, command issuance, monitoring loops, and security policies to create a closed loop for training, deployment, and operation.

[0020] Furthermore, the construction of a digital twin environment corresponding to the real port, and the completion of basic configurations for data, communication, and parameters, specifically includes:

[0021] Import internal and external container truck operation records, equipment operation logs, road traffic information, and gate reservation fulfillment status from the terminal operation system and gate reservation platform. Complete cleaning, time sequence alignment, and tag verification to generate initial task sets, equipment sets, and vehicle sets, and retain key features.

[0022] Based on the storage yard zoning, berth layout, gate location, and access to social roads, a multi-layer topology map is constructed. The nodes of the quay cranes, yard cranes, storage yard areas, gates, and buffer zones are abstracted, and the edges are assigned distance, travel time, congestion coefficient, and safety level to form a road network model that supports two-way traffic of internal and external vehicles, which serves as a digital twin environment model.

[0023] Configure equipment operation capabilities, loading and unloading time distribution, failure rate and maintenance strategy on the digital twin environment model, set parameters such as internal and external truck speed, acceleration and deceleration, load limit, and shift system, and inject external truck reservation rules, gate release rhythm, queuing logic and random disturbances.

[0024] Establish interface protocols with the terminal operation system, gate reservation platform, vehicle terminal and yard control system, and configure message encoding, time synchronization and abnormal retransmission strategies;

[0025] Set the scheduling cycle, rolling window, vehicle load limit, task priority threshold and device concurrency capability, initialize the learning rate, batch size, target network synchronization cycle and reward weight of the deep reinforcement learning model, and pre-set parameter templates corresponding to different operational objectives.

[0026] Furthermore, the step of constructing a dynamic action set for the currently scheduled vehicle and filtering out all candidate actions that violate load, time window, process, or equipment state constraints through a masking mechanism specifically includes:

[0027] Set global action index set ,in The maximum number of devices allowed to participate in the decision-making process simultaneously;

[0028] Based on the current set of executable tasks or dockable devices for the vehicle to be decided, fill the action index sequentially according to a preset order. to This enables candidate action mapping.

[0029] For each candidate action after mapping Check vehicle-related constraints;

[0030] Based on the task status and the status of the associated equipment, check the task scheduling identifier, arrival time window, completion status of previous tasks, whether the equipment is in a disabled / faulty state, and whether the queue is overloaded.

[0031] Construct a mask vector of the same length as the global action index set. Assigning values ​​to legal actions Illegal action assignment .

[0032] Furthermore, the construction of the structure-aware policy network and value network for truck scheduling, and the performance of advantage-weighted updates and multi-day rolling training, specifically includes:

[0033] The task set, vehicle set, and equipment set are respectively input into an encoder with shared parameters to achieve self-attention modeling of set-type inputs. The encoder outputs the element embedding and summary vector of each set, and highlights the embedding of the vehicle to be decided.

[0034] For the candidate task / device mapped to the action index, the context is formed by combining the local embedding of the candidate task / device with the current vehicle embedding and the summary vector of each set. A compatible representation is obtained through cross attention, and an action score is generated through the pointer-type output layer. The action score is multiplied with the mask and then normalized by Softmax to realize the scheduling policy output.

[0035] For the core indicators, a distributed value head is used to output state value estimation. The core indicators include vehicle waiting time, equipment idle time and empty driving distance.

[0036] Advantage-weighted regression is used to update the scheduling strategy, and the value network is jointly optimized by mean squared error and distributed loss.

[0037] Based on production logs from multiple consecutive days, the system is segmented by date, simulated and replayed daily, and iteratively deployed after regular evaluation to achieve multi-day rolling training.

[0038] Furthermore, the construction of a composite reward system, which comprehensively measures vehicle efficiency, equipment utilization, empty mileage, and task completion performance, specifically includes:

[0039] Design a reward function component, which includes vehicle and task waiting time, idle distance increment, equipment idle and queuing penalty, constraint conflict cost and throughput reward.

[0040] Based on the reward function components and reward weights, construct the reward function and configure the reward weights;

[0041] We introduce a latent function based on historical high-quality scheduling samples to shape the immediate reward;

[0042] A flexible balance can be achieved between different operational objectives through weight adjustment or phase switching.

[0043] Furthermore, the reward function is expressed as follows:

[0044] ;

[0045] Where t is the scheduling period, Waiting time for vehicles and tasks, For the increase in empty driving distance, Penalties for idle equipment and queues. To constrain conflict costs, For throughput rewards, For reward weighting.

[0046] Furthermore, the construction of the interface solution, which includes data interfaces, command issuance, monitoring loops, and security policies, specifically includes:

[0047] Establish standardized interfaces with the terminal operation system, video surveillance system, quay crane / yard crane PLC system, and vehicle positioning system to obtain task information, vehicle status, equipment status, and gate operation records according to a unified timestamp;

[0048] The collected real-time data is mapped to a unified state space, generating three sets: tasks, vehicles, and equipment, which are then pushed to the strategy reasoning service.

[0049] Deploy the policy network inference service, obtain the state and action mask according to the set decision cycle, and output scheduling instructions;

[0050] Real-time monitoring of core indicators; when core indicators exceed thresholds or illegal actions are frequently triggered by the mask, alarms are generated and detailed logs are recorded. The core indicators include vehicle waiting time, equipment idle time and empty driving distance.

[0051] Establish a model repository and version control mechanism to complete offline evaluation and gray-scale verification before going live;

[0052] The system continuously collects the latest job data during online operation and periodically triggers the training pipeline.

[0053] The second objective of this invention can be achieved by adopting the following technical solution:

[0054] A container terminal truck scheduling device based on deep reinforcement learning, the device comprising:

[0055] The building unit is used to build a digital twin environment model corresponding to the real port and to complete the basic configuration of data, communication and parameters;

[0056] The first building unit is used to build a unified state space for reinforcement learning agents to use based on the digital twin environment model. The state space simultaneously covers the task set, the vehicle set, and the device set.

[0057] The second construction unit is used to build a dynamic action set for the vehicle to be dispatched based on a unified state space, and to filter out all candidate actions that violate the constraints of load, time window, process or equipment state through a masking mechanism.

[0058] The third building unit is used to construct a structure-aware policy network and a value network for truck scheduling based on a unified state space and action mask, and to perform advantage-weighted updates and multi-day rolling training.

[0059] The fourth building unit is used to construct composite rewards based on a unified state space, which comprehensively measures vehicle efficiency, equipment utilization, empty driving mileage and task completion performance.

[0060] The fifth building unit is used to construct an interface solution that includes data interfaces, command issuance, monitoring loops, and security policies, so that training, deployment, and operation form a closed loop.

[0061] The third objective of this invention can be achieved by adopting the following technical solution:

[0062] A computer device includes a processor and a memory for storing processor-executable programs, wherein when the processor executes the programs stored in the memory, it implements the above-described container terminal truck scheduling method.

[0063] The fourth objective of this invention can be achieved by adopting the following technical solution:

[0064] A computer-readable storage medium storing a program that, when executed by a processor, implements the above-described container terminal truck scheduling method.

[0065] The present invention has the following advantages over the prior art:

[0066] This invention performs unified modeling and collaborative decision-making for internal and external container trucks at the scheduling level, avoiding backflow and congestion between the gate and the yard; through action mask constraints and rolling time window pruning, it ensures that scheduling instructions meet hard constraints such as load, safety, and time window and have real-time performance; it introduces a multi-objective reward function, which can balance waiting time, empty mileage, equipment utilization, energy consumption, and throughput; and it provides a closed-loop architecture from simulation training to online deployment and continuous iteration, with engineering implementation capabilities. Attached Figure Description

[0067] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0068] Figure 1 This is a flowchart of the container terminal truck scheduling method based on deep reinforcement learning according to Embodiment 1 of the present invention.

[0069] Figure 2 This is a schematic diagram of the deep reinforcement learning model training in Embodiment 1 of the present invention.

[0070] Figure 3 This is a model training reward curve diagram of Embodiment 2 of the present invention.

[0071] Figure 4 This is a timing diagram of the scheduling service and the dock operation system call in Embodiment 2 of the present invention.

[0072] Figure 5 This is a structural block diagram of the container terminal truck scheduling device based on deep reinforcement learning according to Embodiment 3 of the present invention.

[0073] Figure 6 This is a structural block diagram of the computer device according to Embodiment 4 of the present invention. Detailed Implementation

[0074] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0075] Example 1:

[0076] like Figure 1 As shown, this embodiment provides a container terminal truck scheduling method based on deep reinforcement learning. This method addresses the complexity, dynamism, and multi-objective optimization requirements of container terminal operations. By establishing a simulation environment and constructing a unified state and action space, it utilizes deep reinforcement learning algorithms to adaptively learn truck scheduling strategies. This achieves coordinated scheduling of internal and external trucks, overall optimization of task chains, and prediction and avoidance of equipment resource conflicts. It is mainly implemented through a truck scheduling system and includes the following steps:

[0077] S101. Build a digital twin environment model corresponding to the real port and complete the basic configuration of data, communication and parameters.

[0078] Before proceeding with strategy training and online scheduling, this embodiment first performs an overall initialization of the truck scheduling system, builds a digital twin environment model corresponding to the real terminal, and completes the basic configuration of data, communication, and parameters to ensure that each module has the ability to operate collaboratively. This allows the constraints of internal and external trucks, loading and unloading equipment, and road network to be accurately characterized and reproduced in a unified model. Specifically, this includes:

[0079] S1011, System Framework Loading: Deploy the core scheduling program to the scheduling center server or cloud cluster, load modules such as data acquisition, status construction, strategy decision-making, simulation training, human-computer interaction, and log auditing, and simultaneously verify the network connectivity with subsystems such as gate, yard control, and traffic management.

[0080] S1012. Importing historical data: Import internal and external container truck operation records, equipment operation logs, road traffic information, and gate reservation fulfillment status from data sources such as the terminal operation system and gate reservation platform. Complete cleaning, time sequence alignment, and tag verification to generate the initial task set T, equipment set M, and vehicle set V, and retain key features such as external container truck reservation defaults and queue lengths.

[0081] S1013. Terminal Digital Twin Modeling: Based on the yard zoning, berth layout, gate location, and access to social roads, construct a multi-layer topology map, abstract nodes for quay cranes, yard cranes, yard areas, gates, and buffer zones, and assign distance, travel time, congestion coefficient, and safety level to the edges to form a road network model that supports two-way traffic of internal and external vehicles, serving as a digital twin environment model.

[0082] S1014. Configure equipment operation capacity, loading and unloading time distribution, failure rate and maintenance strategy on the digital twin environment model, set parameters for internal and external truck speed, acceleration and deceleration, load limit, and shift system, and inject external truck reservation rules, gate release rhythm, queuing logic and random disturbances.

[0083] S1015. Communication and Data Bus Initialization: Establish interface protocols with the terminal operation system, gate reservation platform, vehicle terminal and yard control system, configure message encoding, time synchronization and abnormal retransmission strategies to ensure that the status construction module receives vehicle location, task updates and equipment status in real time, and can issue instructions to internal and external container truck fleets and obtain execution feedback through the strategy decision module.

[0084] S1016. Parameter and Strategy Initialization: Set business parameters such as scheduling cycle, rolling window, vehicle load limit, task priority threshold, and device concurrency capability. Initialize hyperparameters such as learning rate, batch size, target network synchronization cycle, and reward weight of the deep reinforcement learning model, and pre-set parameter templates corresponding to different operational goals to lay the foundation for subsequent training and deployment.

[0085] S102. Based on the digital twin environment model, construct a unified state space for use by reinforcement learning agents.

[0086] After the system initialization in step S101 is completed, a unified state space S is constructed for the reinforcement learning agent to use. This state space also covers the task set. Vehicle assembly (including internal and external container trucks) and equipment collection ,set up This represents the number of currently schedulable tasks. The number of cards collected on site. Given the number of in-service loading and unloading equipment, then:

[0087] ;

[0088] Task Collection , This includes the following variables:

[0089]

[0090] Table 1 Task Status Definition

[0091] The device number uses One-Hot coding, which satisfies... ,in Indicates the number of devices.

[0092] vehicle assembly , This includes the following variables:

[0093]

[0094] Table 2 Truck Status Definition

[0095] The device number uses One-Hot coding, which satisfies... ,in Indicates the number of trucks.

[0096] Equipment Collection , This includes the following variables:

[0097]

[0098] Table 3 Equipment Status Definition

[0099] The device number uses One-Hot coding, which satisfies... ,in Indicates the number of devices.

[0100] The aforementioned state vectors are refreshed in each rolling scheduling cycle and concatenated into a unified state after normalization or encoding. As input to the policy network, it ensures that the model can simultaneously perceive real-time changes in internal and external trucks, gate reservations, equipment queues, and task links.

[0101] S103. Based on the unified state space, construct a dynamic action set for the currently scheduled vehicle, and filter out all candidate actions that violate the constraints of load, time window, process or equipment state through a masking mechanism.

[0102] This embodiment, based on a unified state space, constructs a dynamic action set for the currently scheduled vehicle and uses a masking mechanism to filter out all candidate actions that violate load, time window, process, or equipment state constraints. This ensures that the policy network searches only within the feasible domain, thereby maintaining the stability and security of online decision-making. Specifically, it includes:

[0103] S1031, Action Index Initialization: Pre-set the global action index set. ,in The maximum number of devices that the system allows to participate in decision-making simultaneously; index The remaining indexes are reserved as "hold and wait" actions, while the rest are used as placeholders for tasks or devices, ensuring that the action space dimensions remain constant throughout the training and deployment process.

[0104] S1032, Candidate Action Mapping: For the current vehicle to be decided Based on its set of executable tasks or set of dockable devices, the action index is populated sequentially in a preset order (such as priority, queue order, or distance from the vehicle). to Fill in any missing spaces with empty placeholders.

[0105] S1033, Vehicle Side Constraint Verification: For each mapped candidate action... Check vehicle-related constraints, including whether the remaining load capacity meets mission requirements, whether the vehicle is in a phase where it can accept new missions, whether the vehicle type or energy attributes match mission requirements, and whether the external truck can be booked within the remaining time window. The process of entering and queuing must be completed within the designated time; any action that fails to meet any of the conditions is marked as infeasible.

[0106] S1034. Task and Equipment Constraint Verification: Based on the task status and the status of the equipment to which it belongs, verify the task's schedulable identifier, arrival time window, completion status of preceding operations, whether the equipment is in a disabled / faulty state, and whether the queue exceeds the limit; if it involves two-stage connection, it is also necessary to confirm that the target equipment and yard resources have been reserved, and actions that violate constraints are also marked as infeasible.

[0107] S1035, Mask Generation and Probability Normalization: Constructing a mask vector of the same length as the global action index set A. Assigning values ​​to legal actions Illegal action assignment Action scores output by the policy network After masking and Softmax normalization, we obtain:

[0108] ;

[0109] when In such cases, the system triggers a backup process (such as waiting or manual intervention).

[0110] S1036, Constraint Extension and Audit Log: The masking mechanism supports the extension of rules such as driver shift change, hazardous materials management, and weather-related road closures according to operational needs; the reasons for masked actions are recorded to facilitate log auditing and subsequent strategy retraining, ensuring the traceability and explainability of dispatching decisions.

[0111] S104. Based on a unified state space and action mask, construct a structure-aware policy network and a value network for truck scheduling, and perform advantage-weighted updates and multi-day rolling training.

[0112] like Figure 2 As shown, based on a unified state space and action mask, a structure-aware policy network and a value network for truck scheduling are constructed. Combined with advantage-weighted updates and multi-day rolling training, the model can continuously improve decision-making quality in complex operational data. Specifically, this includes:

[0113] S1041, Encoder (Shared Encoder): A Set Transformer encoder is adopted. The task set, vehicle set, and equipment set are respectively input into the Set Transformer encoder with shared parameters to realize self-attention modeling of set inputs. The encoder outputs the element embedding and summary vector of each set, and highlights the embedding of the vehicle to be decided, so that the network can grasp key information such as its load, operation stage, and task chain.

[0114] S1042, Pointer-type Mask Strategy Header (Strategy Network Output Header): For the candidate task / device mapped to the action index, the context is formed by combining its local embedding with the current vehicle embedding and the summary vector of each set. A compatible representation is obtained through cross-attention, and an action score is generated through the pointer-type output layer. After multiplying the action score with the mask, it is normalized by Softmax. Probabilities are assigned only to legal actions to achieve accurate scheduling strategy output.

[0115] S1043, Value Network and Multi-Objective Evaluation (Value Network Output Head): The value network shares the same encoder front end and focuses on evaluating the impact of scheduling decisions on core indicators such as vehicle waiting time, equipment idle time, and empty driving distance; it adopts a distributed value head to output state value estimates, providing a stable advantage benchmark for the strategy.

[0116] S1044, Advantage-Weighted Policy Update: The scheduling policy is updated using advantage-weighted regression, with the advantage obtained from distributed value estimation and n-step reward calculation; illegal actions masked by the mask do not participate in gradient updates, ensuring that training converges within the actionable action domain; the value network is jointly optimized using mean squared error and distributed loss to maintain numerical stability.

[0117] S1045, Multi-day Rolling Training: Training data comes from production logs over several consecutive days, segmented by date, simulated and replayed daily, and iteratively deployed after periodic evaluation, forming a closed loop of "multi-day data → strategy training → offline evaluation → redeployment". During training, key indicators such as vehicle waiting time, empty driving distance, and equipment utilization are monitored in real time, and hyperparameters such as learning rate, number of attention heads, and dominant temperature coefficient are adjusted as necessary.

[0118] Through the above design, the policy network can fully understand the coupling relationship between tasks, vehicles and equipment under mask constraints, the value network provides a reliable reference for multi-objective optimization, and multi-day rolling training ensures the model's long-term adaptability to fluctuations in real operations.

[0119] S105. Based on a unified state space, construct a composite reward system to comprehensively measure vehicle efficiency, equipment utilization, empty mileage, and task completion performance.

[0120] This embodiment constructs a composite reward based on a unified state space, comprehensively measuring vehicle efficiency, equipment utilization, empty mileage, and task completion performance. This facilitates the formation of a clear optimization orientation for the strategy during training, specifically including:

[0121] S1051. Reward Function Component Design: Let the reward function components of the scheduling period t be the vehicle waiting time and the task waiting time, respectively. Incremental empty driving distance Equipment idleness and queuing penalties Constraints on conflict costs Throughput Rewards The roles of each reward function component are as follows:

[0122] Vehicle and mission waiting time Vehicle waiting time Task waiting time polymerization;

[0123] Increment of empty driving distance : The distance traveled during the mission's empty run Cumulative distance with vehicles Calculation of the change;

[0124] Equipment idle and queue penalties Based on equipment idle time Number of vehicles in queue Number of queued tasks Combined to obtain;

[0125] Constraint Conflict Costs When the task is disabled, the flag is disabled. Or device disabled indicator Record the corresponding penalty when triggered;

[0126] Throughput Rewards Provide positive incentives when tasks are completed.

[0127] S1052. Construct the reward function based on the reward function components and reward weights, as follows:

[0128] ;

[0129] in, For reward weighting.

[0130] S1053, Reward Weight Configuration and Operation Adjustment: By default, the reward weight configuration and operation adjustment are as follows: , The primary focus is on reducing vehicle waiting times and equipment idle time; improving efficiency during peak ship berthing and departure periods. To ensure throughput efficiency; if it is necessary to suppress idling or energy consumption, the capacity can be increased. The reward weight parameters can be adjusted in real time during runtime and recorded in the training log.

[0131] S1054, Reward Shaping and n-Step Reward: To alleviate the sparsity problem in the early stages of training, a latent function based on historical high-quality scheduling samples is introduced. Shaping instant rewards:

[0132] ;

[0133] Advantage estimation combined with n-step returns allows the strategy to focus on both short-term gains and phased goals.

[0134] S106. Construct a solution that includes data interfaces, command issuance, monitoring loops, and security policies to create a closed loop for training, deployment, and operation.

[0135] To enable the algorithm model to be deployed in the dock production environment, this embodiment requires the construction of a system integration solution that includes data interfaces, command issuance, monitoring loops, and security policies, creating a closed loop between training, deployment, and operation. Specifically, this includes:

[0136] S1061, Data Acquisition Bus: Establishes standardized interfaces with terminal operation systems, video surveillance systems, quay crane / yard crane PLC systems, vehicle positioning systems, etc., to acquire task information, vehicle status, equipment status, and gate operation records according to a unified timestamp; the data is cleaned by the ETL module and then written to the online cache and training data warehouse.

[0137] S1062, State Construction Service: Maps the collected real-time data to a unified state space, generates three sets of tasks, vehicles, and equipment, and pushes them to the policy inference service; at the same time, it writes historical data to disk to provide original samples for subsequent rolling training.

[0138] S1063, Strategy Reasoning and Command Scheduling: Deploy strategy network reasoning services, obtain status and action masks according to the set decision cycle, and output scheduling commands; the scheduling commands are sent to the internal truck fleet scheduling terminal and equipment control system through the scheduling platform, and the execution receipt is recorded.

[0139] S1064, Monitoring and Auditing Module: Real-time monitoring of core indicators such as vehicle waiting time, equipment idle time, and task completion rate; when core indicators exceed the threshold or illegal actions are frequently triggered by the mask, alarms are generated and detailed logs are recorded, supporting operation auditing and policy backtracking.

[0140] S1065. Strategy Version Management and Rollback: Establish a model repository and version control mechanism to complete offline evaluation and gray-scale verification before going live; in case of anomalies, it can be quickly rolled back to a historical stable version, and a manual scheduling mode is retained as a backup guarantee.

[0141] S1066, Continuous Iteration and Synchronization: During online operation, the latest operation data is continuously collected, the training pipeline is periodically triggered, the model is updated and verified in the gray-scale environment, and then synchronized to the production system to achieve a closed-loop iteration of "training-deployment-operation".

[0142] The complete simulation and training algorithm pseudocode is as follows:

[0143]

[0144] Through the above-mentioned integration scheme, the scheduling algorithm and the terminal information system are tightly coupled, which not only ensures the executability of real-time decision-making, but also provides stable engineering support for continuous optimization.

[0145] Example 2:

[0146] This example is an inland river container terminal in the Pearl River Delta region, with an annual throughput of approximately 800,000 TEUs. It owns 27 container trucks (19 diesel and 8 new energy), and is equipped with 9 shore-side machines, 42 yard cranes, and basic information systems such as TOS, vehicle-mounted terminals, and wireless private networks.

[0147] S1. System Initialization and Environment Setup

[0148] It connects to interfaces such as TOS operation logs, shore / yard crane PLC signals, truck GPS tracks, gate reservations, and road monitoring to obtain 60 consecutive days of production data, and completes time synchronization, missing data completion, and anomaly removal.

[0149] Based on the layout of berths, storage yards and inland waterway roads, a road network of 52 nodes and 118 directed edges is constructed; loading vessel plans, gate release rhythms and external truck reservation records are also recorded.

[0150] The digital twin simulation is configured with parameters such as the operating capacity of 9 shore-side machines, 42 yard cranes, and 27 container trucks, vehicle speed of 23 km / h, load limit, and driver scheduling, and random events such as road congestion are inserted.

[0151] Establish a unified data bus with TOS / PLC / vehicle terminal, using UTC+8 time base and Protobuf encoding, and control the communication delay within 400 ms.

[0152] Initialization scheduling period 12s, rolling window 36s, reward weight =1.0、 =0.7、 =0.8、 =1.1、 =1.4; set the total training steps to 5,000,000, batch size to 64, n-step reward n=5, and learning rate to 2.5×10. -4 And enable experience replay capacity of 120,000.

[0153] S2. Environment Modeling and State Definition

[0154] The task status includes 14 items such as waiting time, cumulative / empty driving distance, affiliated device, priority, schedulable identifier, task type, and queue length; the maximum number is filled with 300.

[0155] The vehicle status includes 14 items such as vehicle waiting time, remaining / cumulative driving distance, load status, operation stage, queue sign, and number of serviceable devices; the vehicle limit is set at 30.

[0156] The equipment status record includes 12 items such as idle time, queued vehicles / tasks, disabled flag, currently serving vehicles, and operation stage; the equipment limit is set to 20.

[0157] All continuous quantities are normalized, and discrete quantities are encoded using One-Hot encoding, forming a unified typical observation space.

[0158] S3, Action Space and Masking Mechanism

[0159] There are 64 preset action indices (index 0 indicates hold and wait, and 1 to 63 correspond to placeholders for all equipment / tasks at the dock).

[0160] Each round of decision-making maps candidate tasks in the device queue to corresponding indices; if a device has no tasks to be assigned, its index is masked.

[0161] The mask simultaneously verifies constraints such as load, operation stage, equipment disabling, yard capacity, shipping schedule window, and gate cycle time. Illegal actions are set to 0, and the triggering reason is recorded.

[0162] S4, Policy Networks and Learning Algorithms

[0163] The task, vehicle, and device sets in the state input are processed through two layers of multi-head self-attention encoding and ensemble pooling to extract structured features, which are then embedded into a joint input policy network along with the vehicle to be decided.

[0164] The policy output adopts a pointer-based structure, and after multiplying with the mask, the Softmax method obtains the probability of effective actions. The value network uses a shared encoder and distributed value output, and the advantage estimation adopts an n-step reward (n=5).

[0165] Learning rate 2.5 × 10 -4 Batch size 64, discount factor 0.99, advantage λ=0.95, cutoff threshold 0.2, entropy coefficient 0.01, gradient clipping 0.5.

[0166] S5, Reward Function and Multi-Objective Optimization

[0167] Instant rewards are calculated according to the following formula:

[0168] .

[0169] Added conflict penalty 3.0 to masking violations. Higher priority for emergency task assignment. Prioritize its execution; increase [efforts] when shipping peaks and the risk of port congestion increases. Strengthen during off-peak hours To reduce empty runs.

[0170] S6, Training, Deployment, and Continuous Iteration

[0171] Training was performed in a digital twin environment by replaying 30 days of operational data for 60 epochs, accumulating 3×10^6 epochs. 6 Step 1: Monitor metrics such as reward, mask trigger rate, and value loss. After the model converges, it enters the deployment phase. The reward curve is shown below. Figure 3 As shown.

[0172] Gray-scale deployment: Select 9 container trucks for gray-scale operation for 7 days, retain manual strategy comparison to ensure no conflicts and no exceeding the window limit; after verification, full deployment is carried out.

[0173] Continuous iteration: After going live, replay the newly added logs for training for 8-10 epochs every day at midnight, and hot update after verification in the gray-scale environment;

[0174] Statistical results after 45 days of operation: the average vehicle waiting time decreased from 15.8 min to 11.3 min, a decrease of 28.5%; the average daily empty driving distance decreased by 21.2%; the waiting time of the quay crane decreased by 19.6%; and the on-time completion rate of the task increased to 97.4%.

[0175] The timing diagram for the scheduling service and the dock operation system call in this embodiment is as follows: Figure 4 As shown, this embodiment demonstrates that in the Pearl River Delta inland river terminal environment with a capacity of 800,000 TEUs, it can be stably implemented by combining real training parameters, achieving the goals of improving efficiency, reducing energy consumption, and keeping safety constraints under control.

[0176] It should be noted that although the above-described method operations are depicted in a specific order in the accompanying drawings, this does not require or imply that these operations must be performed in that specific order, or that all of the illustrated operations must be performed to achieve the desired result. On the contrary, the order of execution of the depicted steps can be changed. Additionally or alternatively, certain steps may be omitted, multiple steps may be combined into one step, and / or one step may be broken down into multiple steps.

[0177] Example 3:

[0178] like Figure 5 As shown, this embodiment provides a container terminal truck scheduling device based on deep reinforcement learning. The device includes a construction unit 501, a first construction unit 502, a second construction unit 503, a third construction unit 504, a fourth construction unit 505, and a fifth construction unit 506. The specific descriptions of each unit are as follows:

[0179] Unit 501 is used to build a digital twin environment model corresponding to the real port and to complete the basic configuration of data, communication and parameters.

[0180] The first construction unit 502 is used to construct a unified state space for reinforcement learning agents based on the digital twin environment model. The state space simultaneously covers the task set, the vehicle set, and the device set.

[0181] The second construction unit 503 is used to construct a dynamic action set for the currently scheduled vehicle based on a unified state space, and to filter out all candidate actions that violate the constraints of load, time window, process or equipment state through a masking mechanism.

[0182] The third building unit 504 is used to build a structure-aware policy network and a value network for truck scheduling based on a unified state space and action mask, and to perform advantage-weighted updates and multi-day rolling training.

[0183] The fourth building unit 505 is used to build a composite reward based on a unified state space, which comprehensively measures vehicle efficiency, equipment utilization, empty driving mileage and task completion performance.

[0184] The fifth building unit 506 is used to build an interface solution that includes data interfaces, command issuance, monitoring loops and security policies, so that training, deployment and operation form a closed loop.

[0185] The specific implementation of each module in this embodiment can be found in Embodiment 1 above, and will not be repeated here. It should be noted that the device provided in this embodiment is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional units as needed, that is, the internal structure can be divided into different functional units to complete all or part of the functions described above.

[0186] Example 4:

[0187] This embodiment provides a computer device, such as... Figure 6 As shown, it includes a processor 602, a memory, an input device 603, a display device 604, and a network interface 605 connected via a device bus 601. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium 606 and internal memory 607. The non-volatile storage medium 606 stores operating devices, computer programs, and a database. The internal memory 607 provides an environment for the operation of the operating devices and computer programs in the non-volatile storage medium. When the processor 602 executes the computer program stored in the memory, it implements the container terminal truck scheduling method of Embodiment 1 described above, as follows:

[0188] A digital twin environment model corresponding to the real terminal is constructed, and the basic configuration of data, communication, and parameters is completed. Based on the digital twin environment model, a unified state space for reinforcement learning agents is constructed, which simultaneously covers the task set, vehicle set, and equipment set. On the basis of the unified state space, a dynamic action set is constructed for the vehicles to be dispatched, and all candidate actions that violate the constraints of load, time window, process, or equipment state are filtered out through a masking mechanism. On the basis of the unified state space and action masking, a structure-aware policy network and a value network for truck dispatching are constructed, and advantage-weighted updates and multi-day rolling training are performed. On the basis of the unified state space, a composite reward is constructed to comprehensively measure vehicle efficiency, equipment utilization, empty mileage, and task completion performance. A docking scheme including data interface, instruction issuance, monitoring loop, and safety policy is constructed to form a closed loop for training, deployment, and operation.

[0189] Storage media includes both computer storage media and communication media, with communication media including any media that can be enabled to transfer a computer program from one place to another. Storage media can be any usable media accessible by a computer. By way of example and not limitation, non-transitory media can include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, magnetic disk storage devices or other magnetic storage devices, or any other media that can be used to store desired program code in the form of instructions or data structures and is accessible by a computer. Moreover, any connection may be properly referred to as computer-readable media. As used herein, disks and optical discs include compact optical discs (CDs), laser discs, optical discs, digital versatile discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media. In addition, the operation of a method or algorithm may reside as any or any combination or set of code and instructions on machine-readable or computer-readable media that can be incorporated into a computer program product.

[0190] Processors include general-purpose single-chip or multi-chip processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. General-purpose processors may be microprocessors or any conventional processor, controller, microcontroller, or state machine. Processors may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. In some implementations, specific processes and methods may be performed by circuitry specifically designed for a given function.

[0191] Example 5:

[0192] This embodiment provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the container terminal truck scheduling method of Embodiment 1 above, as follows:

[0193] A digital twin environment model corresponding to the real terminal is constructed, and the basic configuration of data, communication, and parameters is completed. Based on the digital twin environment model, a unified state space for reinforcement learning agents is constructed, which simultaneously covers the task set, vehicle set, and equipment set. On the basis of the unified state space, a dynamic action set is constructed for the vehicles to be dispatched, and all candidate actions that violate the constraints of load, time window, process, or equipment state are filtered out through a masking mechanism. On the basis of the unified state space and action masking, a structure-aware policy network and a value network for truck dispatching are constructed, and advantage-weighted updates and multi-day rolling training are performed. On the basis of the unified state space, a composite reward is constructed to comprehensively measure vehicle efficiency, equipment utilization, empty mileage, and task completion performance. A docking scheme including data interface, instruction issuance, monitoring loop, and safety policy is constructed to form a closed loop for training, deployment, and operation.

[0194] It should be noted that the computer-readable storage medium in this embodiment can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0195] In this embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this embodiment, the computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable program. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable storage medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable storage medium can be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0196] The computer-readable storage medium described above can be used to write computer programs for executing this embodiment in one or more programming languages ​​or combinations thereof. These programming languages ​​include object-oriented programming languages—such as Java, Python, and C++—and conventional procedural programming languages—such as C or similar programming languages. The program can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0197] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Unless otherwise specified, an element defined by the phrase "comprising..." or "including..." does not exclude the presence of additional elements in the process, method, article, or terminal device that includes said element. Additionally, in this document, "greater than," "less than," "exceeding," etc., are understood to exclude the stated number; "above," "below," "within," etc., are understood to include the stated number.

[0198] In summary, this invention provides unified modeling and collaborative decision-making for internal and external container trucks at the scheduling level, avoiding backflow and congestion between the gate and the yard; through action mask constraints and rolling time window pruning, it ensures that scheduling instructions meet hard constraints such as load, safety, and time window and have real-time performance; it introduces a multi-objective reward function, which can balance waiting time, empty mileage, equipment utilization, energy consumption, and throughput; and it provides a closed-loop architecture from simulation training to online deployment and continuous iteration, with engineering implementation capabilities.

[0199] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope disclosed in the present invention, based on the technical solution and inventive concept of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A container terminal truck scheduling method based on deep reinforcement learning, characterized in that, The method includes: Build a digital twin environment model corresponding to the real port, and complete the basic configuration of data, communication and parameters; Based on the digital twin environment model, a unified state space is constructed for use by reinforcement learning agents, which simultaneously covers the task set, vehicle set, and device set; Based on a unified state space, a dynamic action set is constructed for the vehicle to be dispatched, and a masking mechanism is used to filter out all candidate actions that violate load, time window, process or equipment state constraints. Based on a unified state space and action mask, a structure-aware policy network and a value network for truck scheduling are constructed, and advantage-weighted updates and multi-day rolling training are performed. Based on a unified state space, a composite reward system is constructed to comprehensively measure vehicle efficiency, equipment utilization, empty mileage, and task completion performance. Build a solution that integrates data interfaces, command issuance, monitoring loops, and security policies to create a closed loop for training, deployment, and operation.

2. The container terminal truck scheduling method according to claim 1, characterized in that, The construction of a digital twin environment corresponding to the real port, and the completion of basic configuration of data, communication and parameters, specifically includes: Import internal and external container truck operation records, equipment operation logs, road traffic information, and gate reservation fulfillment status from the terminal operation system and gate reservation platform. Complete cleaning, time sequence alignment, and tag verification to generate initial task sets, equipment sets, and vehicle sets, and retain key features. Based on the storage yard zoning, berth layout, gate location, and access to social roads, a multi-layer topology map is constructed. The nodes of the quay cranes, yard cranes, storage yard areas, gates, and buffer zones are abstracted, and the edges are assigned distance, travel time, congestion coefficient, and safety level to form a road network model that supports two-way traffic of internal and external vehicles, which serves as a digital twin environment model. Configure equipment operation capabilities, loading and unloading time distribution, failure rate and maintenance strategy on the digital twin environment model, set parameters such as internal and external truck speed, acceleration and deceleration, load limit, and shift system, and inject external truck reservation rules, gate release rhythm, queuing logic and random disturbances. Establish interface protocols with the terminal operation system, gate reservation platform, vehicle terminal and yard control system, and configure message encoding, time synchronization and abnormal retransmission strategies; Set the scheduling cycle, rolling window, vehicle load limit, task priority threshold and device concurrency capability, initialize the learning rate, batch size, target network synchronization cycle and reward weight of the deep reinforcement learning model, and pre-set parameter templates corresponding to different operational objectives.

3. The container terminal truck scheduling method according to claim 1, characterized in that, The process of constructing a dynamic action set for the currently scheduled vehicle and filtering out all candidate actions that violate load, time window, process, or equipment state constraints using a masking mechanism specifically includes: Set global action index set ,in The maximum number of devices allowed to participate in the decision-making process simultaneously; Based on the current set of executable tasks or dockable devices for the vehicle to be decided, fill the action index sequentially according to a preset order. to This enables candidate action mapping. For each candidate action after mapping Check vehicle-related constraints; Based on the task status and the status of the associated equipment, check the task scheduling identifier, arrival time window, completion status of previous tasks, whether the equipment is in a disabled / faulty state, and whether the queue is overloaded. Construct a mask vector of the same length as the global action index set. Assigning values ​​to legal actions Illegal action assignment .

4. The container terminal truck scheduling method according to claim 1, characterized in that, The construction of the structure-aware policy network and value network for truck scheduling, and the subsequent advantage-weighted updates and multi-day rolling training, specifically include: The task set, vehicle set, and equipment set are respectively input into an encoder with shared parameters to achieve self-attention modeling of set-type inputs. The encoder outputs the element embedding and summary vector of each set, and highlights the embedding of the vehicle to be decided. For the candidate task / device mapped to the action index, the context is formed by combining the local embedding of the candidate task / device with the current vehicle embedding and the summary vector of each set. A compatible representation is obtained through cross attention, and an action score is generated through the pointer-type output layer. The action score is multiplied with the mask and then normalized by Softmax to realize the scheduling policy output. For the core indicators, a distributed value head is used to output state value estimation. The core indicators include vehicle waiting time, equipment idle time and empty driving distance. Advantage-weighted regression is used to update the scheduling strategy, and the value network is jointly optimized by mean squared error and distributed loss. Based on production logs from multiple consecutive days, the system is segmented by date, simulated and replayed daily, and iteratively deployed after regular evaluation to achieve multi-day rolling training.

5. The container terminal truck scheduling method according to claim 1, characterized in that, The aforementioned composite reward system comprehensively measures vehicle efficiency, equipment utilization, empty mileage, and task completion performance, specifically including: Design a reward function component, which includes vehicle and task waiting time, idle distance increment, equipment idle and queuing penalty, constraint conflict cost and throughput reward. Based on the reward function components and reward weights, construct the reward function and configure the reward weights; We introduce a latent function based on historical high-quality scheduling samples to shape the immediate reward; A flexible balance can be achieved between different operational objectives through weight adjustment or phase switching.

6. The container terminal truck scheduling method according to claim 5, characterized in that, The reward function is expressed as follows: ； Where t is the scheduling period, Waiting time for vehicles and tasks, For the increase in empty driving distance, Penalties for idle equipment and queues. To constrain conflict costs, For throughput rewards, For reward weighting.

7. The container terminal truck scheduling method according to claim 1, characterized in that, The proposed solution includes an interface for data interfaces, command issuance, monitoring loops, and security policies. Specifically, it includes: Establish standardized interfaces with the terminal operation system, video surveillance system, quay crane / yard crane PLC system, and vehicle positioning system to obtain task information, vehicle status, equipment status, and gate operation records according to a unified timestamp; The collected real-time data is mapped to a unified state space, generating three sets: tasks, vehicles, and equipment, which are then pushed to the strategy reasoning service. Deploy the policy network inference service, obtain the state and action mask according to the set decision cycle, and output scheduling instructions; Real-time monitoring of core indicators; when core indicators exceed thresholds or illegal actions are frequently triggered by the mask, alarms are generated and detailed logs are recorded. The core indicators include vehicle waiting time, equipment idle time and empty driving distance. Establish a model repository and version control mechanism to complete offline evaluation and gray-scale verification before going live; The system continuously collects the latest job data during online operation and periodically triggers the training pipeline.

8. A container terminal truck scheduling device based on deep reinforcement learning, characterized in that, The device includes: The building unit is used to build a digital twin environment model corresponding to the real port and to complete the basic configuration of data, communication and parameters; The first building unit is used to build a unified state space for reinforcement learning agents to use based on the digital twin environment model. The state space simultaneously covers the task set, the vehicle set, and the device set. The second construction unit is used to build a dynamic action set for the vehicle to be dispatched based on a unified state space, and to filter out all candidate actions that violate the constraints of load, time window, process or equipment state through a masking mechanism. The third building unit is used to construct a structure-aware policy network and a value network for truck scheduling based on a unified state space and action mask, and to perform advantage-weighted updates and multi-day rolling training. The fourth building unit is used to construct composite rewards based on a unified state space, which comprehensively measures vehicle efficiency, equipment utilization, empty driving mileage and task completion performance. The fifth building unit is used to construct an interface solution that includes data interfaces, command issuance, monitoring loops, and security policies, so that training, deployment, and operation form a closed loop.

9. A computer device comprising a processor and a memory for storing a processor-executable program, characterized in that, When the processor executes the program stored in the memory, it implements the container terminal truck scheduling method according to any one of claims 1-7.

10. A computer-readable storage medium storing a program, characterized in that, When the program is executed by the processor, it implements the container terminal truck scheduling method according to any one of claims 1-7.

Images