Intelligent port scheduling and energy efficiency optimization method based on digital twinning

Abstract

The invention discloses an intelligent port scheduling and energy efficiency optimization method based on digital twinning. The method comprises the steps that S1, three-dimensional point cloud data and sensor data of a target port are acquired; s2, constructing a port digital twin model comprising a quay crane model, an AGV model and a storage yard model; s3, agents are constructed for a quay crane model, an AGV model and a storage yard model in the port digital twinborn model, and a multi-agent reinforcement learning environment is built; s4, training each agent in a multi-agent reinforcement learning environment through a multi-agent reinforcement learning algorithm; s5, mapping the real-time task data and state data of the port to each trained agent to simulate the scheduling of the target port, and generating a port scheduling strategy; and S6, realizing operation scheduling of the quay crane equipment, the AGV equipment and the storage yard equipment of the target port based on the port scheduling strategy. According to the invention, the operation efficiency and the energy efficiency level of the port can be improved, and powerful technical support is provided for sustainable development of the intelligent port.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent port scheduling and energy efficiency optimization, and particularly relates to an intelligent port scheduling and energy efficiency optimization method based on digital twin. Background Art

[0002] With the continuous growth of global trade and the rapid development of the logistics industry, ports, as key nodes connecting ocean transportation and inland transportation, have a crucial impact on the smooth operation of the entire logistics chain in terms of their operation efficiency and energy efficiency level. The concept of intelligent ports has emerged, which uses modern information technologies, Internet of Things, big data, artificial intelligence and other advanced technologies to comprehensively and intelligently transform the production, operation and management of ports, aiming to improve the operation efficiency of ports, reduce operation costs, enhance safety performance, and achieve environmentally friendly development.

[0003] The core of an intelligent port lies in the realization of optimal resource allocation and efficient scheduling, which includes aspects such as the planning of ship entry and exit from the port, the loading, unloading and storage management of goods, and the scheduling and control of in-port transportation equipment. However, despite the remarkable progress made in the construction of intelligent ports, a series of challenges and problems still exist in the actual scheduling and energy efficiency optimization process.

[0004] Traditional port scheduling systems often rely on manual experience and rule-based decision-making methods, which are unable to cope with the complex and changeable port operation environment. Port operations involve multiple links and numerous equipment, such as quay cranes, automated guided vehicles (AGVs), yard equipment, etc., and the coordinated operation between them requires a high degree of coordination and real-time performance. Traditional scheduling methods are difficult to fully consider various dynamic factors, such as weather changes, ship delays, equipment failures, etc., resulting in the lack of flexibility and robustness of scheduling schemes.

[0005] In terms of energy efficiency optimization, the high energy consumption of port equipment has always been an important factor restricting the sustainable development of ports. Quay cranes, AGVs and yard equipment consume a large amount of energy during operation, and due to the uneven distribution of operation tasks and the non-optimal operation of equipment, the energy efficiency is low. Traditional energy efficiency management methods often focus on the energy-saving transformation of single equipment or the optimization of operation parameters, lacking the ability to optimize the overall energy efficiency from the system level.

[0006] To solve the above problems, many new technologies and methods have emerged in recent years. As an emerging digital means, digital twin technology provides new ideas for the intelligent management of ports by constructing a virtual mirror of the physical world. Digital twin technology can reflect the operating status of ports in real time and provide accurate data support for scheduling decisions.

[0007] However, the current application of digital twin technology in port scheduling and energy efficiency optimization is still in the exploratory stage and has many deficiencies. On the one hand, existing digital twin models often focus on simulating the physical structure or equipment status of ports and lack the ability to model and analyze complex dynamic behaviors during port operations. This limits the auxiliary role of digital twin models in scheduling decisions and makes it difficult to fully exploit their potential. On the other hand, although artificial intelligence technologies such as reinforcement learning have achieved certain application results in the field of port scheduling, most of them are concentrated on the scheduling optimization of single equipment or simple scenarios. For complex port systems containing multiple agents (such as quay cranes, AGVs, yard equipment), how to achieve coordinated scheduling and energy efficiency optimization of multiple agents remains an urgent problem to be solved. Summary of the Invention

[0008] Aiming at the deficiencies of the above-mentioned existing technologies, the technical problem to be solved by the present invention is: how to provide a digital-twin-based intelligent port scheduling and energy efficiency optimization method, which effectively solves the current technical problems of intelligent port scheduling and energy efficiency optimization through technical means such as accurately constructing a port digital twin model, designing an intelligent multi-agent system, efficiently training the multi-agent system, real-time mapping and strategy generation, and comprehensively optimizing the operation scheduling of port equipment, improves the operation efficiency and energy efficiency level of the port, and provides strong technical support for the sustainable development of intelligent ports.

[0009] To solve the above technical problems, the present invention adopts the following technical solutions:

[0010] A digital-twin-based intelligent port scheduling and energy efficiency optimization method, comprising:

[0011] S1: Obtain the three-dimensional point cloud data and sensor data of the target port;

[0012] S2: Based on the three-dimensional point cloud data and sensor data of the target port, construct a port digital twin model including a quay crane model, an AGV model, and a yard model;

[0013] S3: Respectively construct agents for the quay crane model, AGV model, and yard model in the port digital twin model, and build a multi-agent reinforcement learning environment;

[0014] S4: Train each agent in the multi-agent reinforcement learning environment through a multi-agent reinforcement learning algorithm;

[0015] S5: Map the real-time task data and status data of the port into the trained agents to simulate the scheduling of the target port and generate a port scheduling strategy;

[0016] S6: Based on the port scheduling strategy, implement the operation scheduling of the quay crane equipment, AGV equipment, and yard equipment of the target port.

[0017] Preferably, in step S1, the three-dimensional point cloud data of the port includes ground point cloud data obtained by a ground laser scanning device on the ground where the port is located, and aerial point cloud data obtained by an unmanned aerial vehicle (UAV) laser scanning device.

[0018] Preferably, in step S1, the sensor data of the port includes data collected by a positioning sensor, a status sensor, and a motion sensor.

[0019] Preferably, in step S2, the port digital twin model is constructed through the following steps:

[0020] S201: Perform preprocessing on the three-dimensional point cloud data and sensor data, including cleaning, denoising, and registration;

[0021] S202: Fuse the preprocessed ground point cloud data and aerial point cloud data to obtain fused point cloud data;

[0022] S202: Perform three-dimensional modeling on the fused point cloud data through the Poisson surface reconstruction algorithm to obtain a three-dimensional point cloud model of the port;

[0023] S203: Extract the feature points and contour lines of the quay crane equipment from the three-dimensional point cloud model of the port, and construct a quay crane model based on the feature points and contour lines of the quay crane equipment through three-dimensional modeling software;

[0024] S204: Construct an AGV model based on the actual size and shape of the AGV equipment through three-dimensional modeling software;

[0025] S205: Extract the contour and height information of the yard from the three-dimensional point cloud model of the port; construct a yard model through three-dimensional modeling software in combination with the types, sizes, and stacking methods of the goods in the yard;

[0026] S206: Fuse the sensor data with the quay crane model, AGV model, and yard model;

[0027] S207: Assemble and integrate the quay crane model, AGV model, and yard model integrated with the sensor data to generate a port digital twin model.

[0028] Preferably, in step S3, the multi-agent reinforcement learning environment is constructed through the following steps:

[0029] S301: Construct a quay crane agent, an AGV agent, and a yard agent;

[0030] S302: Design the state space, action space, and reward function of the quay crane agent;

[0031] The state space is defined as the set of the position of the quay crane, the height of the spreader, the rotation angle of the spreader, and the position of the target cargo;

[0032] The action space is defined as the set of actions that the quay crane can perform;

[0033] The formula of the reward function is expressed as:

[0034] r aq = α 1 · e aq - α 2 · c aq - α 3 · t aq ;

[0035] In the formula: e aq represents the speed at which the quay crane completes the task; c aq represents that the quay crane has a collision; t aq represents that the quay crane operation times out; α 1 , α 2 , α 3 represent weight coefficients;

[0036] S303: Design the state space, action space, and reward function of the AGV agent;

[0037] The state space is defined as the set of the position, speed, direction, and target position of the AGV;

[0038] The action space is defined as the set of actions that the AGV can perform;

[0039] The formula of the reward function is expressed as:

[0040] r agv = β 1 · (1 - t agv ) - β 2 · p agv - β 3 · c agv ;

[0041] In the formula: t agv represents the time required for the AGV to reach the target position; p agv represents the path smoothness; p agv represents that the AGV has a collision; β 1 , β 2 , β 3 represent weight coefficients;

[0042] S304: Design the state space, action space, and reward function of the yard agent;

[0043] The state space is defined as the set of the position, type, quantity of goods in the yard, and the free space in the yard;

[0044] The action space is defined as the set of actions that the yard can perform;

[0045] The formula of the reward function is expressed as:

[0046] r dc = γ 1 · s dc - γ 2 · r dc - γ 3 · c dc ;

[0047] In the formula: s dc represents the yard space utilization efficiency; r dc represents the time required to retrieve goods; c dc represents a collision occurring during yard operations; γ 1 , γ 2 , γ 3 represent weight coefficients;

[0048] S305: Use the port digital twin model as a multi-agent reinforcement learning environment, enabling the quay crane agent, AGV agent, and yard agent to interact and collaborate in the multi-agent reinforcement learning environment;

[0049] S306: Based on the state spaces, action spaces, and reward functions of the quay crane agent, AGV agent, and yard agent, construct the global state space, global action space, and global reward function of the multi-agent reinforcement learning environment, and at the same time define the state transition function;

[0050] The global state space is the set of the state spaces of the quay crane agent, AGV agent, and yard agent;

[0051] The global action space is the set of the action spaces of the quay crane agent, AGV agent, and yard agent;

[0052] The global reward function is the sum of the reward functions of the quay crane agent, AGV agent, and yard agent;

[0053] The formula of the state transition function is expressed as:

[0054] S′ = f(S, A);

[0055] In the formula: S′ represents the new state after state transition; S represents the current state; A represents the current action of the agent; f is the function symbol.

[0056] Preferably, in step S301:

[0057] The quay crane agent is responsible for controlling the operation of the quay crane to complete the cargo handling task;

[0058] The AGV agent is responsible for controlling the movement and path planning of the AGV to complete the cargo transportation task;

[0059] The yard agent is responsible for monitoring and managing the stacking and retrieval of goods in the yard.

[0060] Preferably, in step S4, the training of each agent is completed through the following steps:

[0061] S401: Initialize the port digital twin model in a multi-agent reinforcement learning environment, including the initial states of the quay crane model, AGV model, and yard model;

[0062] S402: Initialize the policy network and value network of the quay crane agent, AGV agent, and yard agent;

[0063] S403: Initialize an experience replay buffer for storing the interaction experiences of each agent in the environment; during training, each time an agent executes a step, its experience is stored in the buffer;

[0064] S404: Update the policies of each agent through the following steps:

[0065] S4041: For each agent, randomly sample a batch of experiences from the experience replay buffer;

[0066] S4042: Use the sampled experiences to calculate the action values under the current policy;

[0067] S4043: Update the policy network parameters of each agent according to the action value gradient ascent to maximize the expected reward;

[0068] S405: Update the value function of each agent through the following steps:

[0069] S4051: For each agent, calculate the target value using the sampled experiences;

[0070] S4052: Use the mean square error between the target value and the value predicted by the current value network as the loss function to update the value network parameters of each agent;

[0071] S406: Regularly copy the parameters of the policy network and value network to the target network of the corresponding agent;

[0072] S407: Repeat steps S404 to S406 until the trained quay crane agent, AGV agent, and yard agent are obtained when the preset training steps or performance metrics are reached.

[0073] Preferably, in step S501, the task data includes cargo handling task data, transportation task data, yard operation task data, and special task requirements;

[0074] The status data includes quay crane status data, AGV status data, yard status data, and port environment status data.

[0075] Preferably, in step S5, the port scheduling strategy is generated through the following steps:

[0076] S501: Obtain the real-time task data and status data of the port;

[0077] S502: Map the quay crane status data and the cargo handling task data to the input space of the quay crane agent; the quay crane agent outputs the operation actions of the quay crane as the quay crane scheduling strategy according to the input data and the policy network;

[0078] S503: Map the AGV status data and the transportation task data to the input space of the AGV agent; the AGV agent outputs the driving path, speed adjustment, and steering information of the AGV device as the AGV scheduling strategy according to the input data and the policy network;

[0079] S504: Map the yard status data and the yard operation task data to the input space of the yard agent; the yard agent outputs the operation instructions of the yard equipment as the yard scheduling strategy according to the input data and the policy network;

[0080] S505: Integrate the quay crane scheduling strategy, the AGV scheduling strategy, and the yard scheduling strategy to generate the port scheduling strategy.

[0081] Preferably, in step S6, the port scheduling strategy is decomposed into specific instructions and sent to the quay crane equipment, the AGV equipment, and the yard equipment for execution respectively.

[0082] Compared with the prior art, the intelligent port scheduling and energy efficiency optimization method based on digital twin in the present invention has the following beneficial effects:

[0083] By obtaining the three-dimensional point cloud data (including ground point cloud and control point cloud) and sensor data of the port, the present invention can comprehensively and accurately capture the physical structure and dynamic information of the port, and the constructed port digital twin model includes a quay crane model, an AGV model, and a yard model. These models are highly consistent with the actual port equipment in terms of geometric shape, motion characteristics, operation logic, etc., providing an accurate virtual environment for subsequent simulation of the scheduling strategy.

[0084] For each key component in the port digital twin model, the present invention respectively designs a quay crane agent, an AGV agent, and a yard agent. These agents can make independent decisions according to their respective tasks and statuses. With the establishment of the multi-agent reinforcement learning environment, these agents can interact and learn in the virtual environment, continuously optimize their own scheduling strategies, and enhance the flexibility and adaptability of port scheduling.

[0085] Through the multi-agent reinforcement learning algorithm, the present invention can efficiently train each agent in a multi-agent reinforcement learning environment, enabling the agent to quickly learn the optimal scheduling strategy. Moreover, the data-driven and feedback mechanisms during the training process ensure the continuous optimization and update of the scheduling strategy, making the generated port scheduling strategy more in line with the actual port operation requirements.

[0086] The present invention can map the real-time task data and status data of the port into each trained agent. By simulating the scheduling process of the target port, a scheduling strategy that conforms to the current port status is generated. Through this real-time mapping and strategy generation method, the port scheduling can respond more promptly to the changes in port operations, achieving dynamic scheduling optimization and improving the port operation efficiency and energy efficiency level. Based on the generated port scheduling strategy, the present invention can comprehensively optimize the operation scheduling of quay cranes, AGVs, and yard equipment in the target port. Through reasonable task allocation, path planning, and resource scheduling, the present invention can reduce the idle time and waiting time of equipment, improve the utilization rate and operation efficiency of equipment, thereby reducing the port operation cost and enhancing the overall energy efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] In order to make the objectives, technical solutions, and advantages of the invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings, where:

[0088] Figure 1 It is a logic block diagram of a digital twin-based intelligent port scheduling and energy efficiency optimization method. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0089] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Components of the embodiments of the present invention described and illustrated in the drawings here can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts fall within the scope of protection of the present invention.

[0090] It should be noted that similar reference numerals and letters denote similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In the description of the present invention, it should be noted that the orientation or positional relationship indicated by the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc. is based on the orientation or positional relationship shown in the figures, or the orientation or positional relationship in which the product of the invention is usually placed during use. It is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the present invention. In addition, the terms "first", "second", "third", etc. are only used for distinguishing descriptions and cannot be understood as indicating or implying relative importance. In addition, terms such as "horizontal" and "vertical" do not mean that the components are required to be absolutely horizontal or hanging, but can be slightly inclined. For example, "horizontal" only means that its direction is more horizontal relative to "vertical", and does not mean that the structure must be completely horizontal, but can be slightly inclined. In the description of the present invention, it should also be noted that unless otherwise clearly specified and limited, the terms "set", "installed", "connected", "connected" should be understood in a broad sense. For example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be directly connected or indirectly connected through an intermediate medium, and it can be the communication inside two elements. For those of ordinary skill in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0091] The following will be further described in detail through specific embodiments:

[0092] Embodiment:

[0093] A method for intelligent port scheduling and energy efficiency optimization based on digital twin is disclosed in this embodiment.

[0094] As Figure 1 shown, the method for intelligent port scheduling and energy efficiency optimization based on digital twin includes:

[0095] S1: Obtain the three-dimensional point cloud data and sensor data of the target port;

[0096] S2: Based on the three-dimensional point cloud data and sensor data of the target port, construct a port digital twin model including quay crane models, AGV models, and yard models;

[0097] S3: Construct agents for the quay crane models, AGV models, and yard models in the port digital twin model respectively, and build a multi-agent reinforcement learning environment;

[0098] S4: Train each agent in a multi-agent reinforcement learning environment through a multi-agent reinforcement learning algorithm;

[0099] In this embodiment, the multi-agent reinforcement learning algorithm is the multi-agent deep deterministic policy gradient (MADDPG) algorithm.

[0100] S5: Map the real-time task data and status data of the port into the trained agents to simulate the scheduling of the target port and generate a port scheduling strategy;

[0101] S6: Implement the operation scheduling of quay cranes, AGVs, and yard equipment at the target port based on the port scheduling strategy.

[0102] By obtaining the three-dimensional point cloud data (including ground point cloud and control point cloud) and sensor data of the port, the present invention can comprehensively and accurately capture the physical structure and dynamic information of the port. Moreover, the constructed port digital twin model includes quay crane models, AGV models, and yard models, and these models are highly consistent with the actual port equipment in terms of geometric shape, motion characteristics, operation logic, etc., providing an accurate virtual environment for subsequent scheduling strategy simulation.

[0103] For each key component in the port digital twin model, the present invention separately designs quay crane agents, AGV agents, and yard agents. These agents can make independent decisions according to their respective tasks and statuses. And with the establishment of the multi-agent reinforcement learning environment, these agents can interact and learn in the virtual environment, continuously optimizing their own scheduling strategies, enhancing the flexibility and adaptability of port scheduling.

[0104] Through the multi-agent reinforcement learning algorithm, the present invention can efficiently train each agent in the multi-agent reinforcement learning environment, enabling the agent to quickly learn the optimal scheduling strategy. And the data-driven and feedback mechanism during the training process ensures the continuous optimization and update of the scheduling strategy, making the generated port scheduling strategy more in line with the actual operation requirements of the port.

[0105] The present invention can map the real-time task data and status data of the port to each trained intelligent agent. By simulating the scheduling process of the target port, a scheduling strategy that conforms to the current port status is generated. Through this real-time mapping and strategy generation method, the port scheduling can respond more promptly to the changes in port operations, realizing dynamic scheduling optimization and improving the port's operation efficiency and energy efficiency level. Based on the generated port scheduling strategy, the present invention can comprehensively optimize the operation scheduling of quay cranes, AGV equipment, and yard equipment at the target port. Through reasonable task allocation, path planning, and resource scheduling, the present invention can reduce the idle time and waiting time of the equipment, improve the utilization rate and operation efficiency of the equipment, thereby reducing the port's operation cost and enhancing the overall energy efficiency.

[0106] To better introduce the technical solution of the present invention, this embodiment will be described through the following several parts.

[0107] I. Port digital twin model

[0108] 1. Three-dimensional point cloud data

[0109] The three-dimensional point cloud data of the port includes the ground point cloud data obtained by the ground laser scanning equipment located on the ground where the port is located, and the aerial point cloud data obtained by the UAV laser scanning equipment.

[0110] In this embodiment, the ground laser scanning equipment (radar) is used to scan the port ground to obtain the three-dimensional coordinate information of ground objects, including docks, roads, building foundations, etc. The UAV laser scanning equipment is used to scan the airspace above the port to obtain the three-dimensional structure information of the whole port, including the tops of buildings, crane structures, yard heights, etc.

[0111] 2. Sensor data

[0112] The sensor data of the port includes the data collected by positioning sensors, status sensors, and motion sensors.

[0113] In this embodiment, the data collected by the positioning sensors includes data from satellite positioning systems such as GPS and Beidou, which are used to determine the real-time positions of equipment such as quay cranes, AGVs, and yards. The data collected by the status sensors includes the data obtained by temperature sensors, humidity sensors, pressure sensors, etc., which are used to monitor the port environment and the operating status of the equipment. The data collected by the motion sensors includes the data obtained by accelerometers, gyroscopes, etc., which are used to capture the motion status of mobile equipment such as AGVs.

[0114] 3. Construction of the port digital twin model

[0115] The port digital twin model is constructed through the following steps:

[0116] S201: Preprocess the 3D point cloud data and sensor data, including cleaning, denoising, and registration;

[0117] S202: Fuse the preprocessed ground point cloud data and aerial point cloud data to obtain fused point cloud data;

[0118] S202: Perform 3D modeling based on the fused point cloud data through the Poisson surface reconstruction algorithm to obtain a (preliminary) 3D point cloud model of the port;

[0119] S203: Extract the feature points and contour lines of the quay crane equipment from the 3D point cloud model of the port, and construct a quay crane model based on the feature points and contour lines of the quay crane equipment through 3D modeling software;

[0120] In this embodiment, the quay crane model includes the structure of the quay crane, moving parts (such as spreaders), and operating states, etc.

[0121] S204: Construct an AGV model based on the actual size and shape of the AGV equipment through 3D modeling software; at the same time, determine the real-time position and movement trajectory of the AGV equipment according to the data collected by the positioning sensor and motion sensor;

[0122] In this embodiment, the AGV model includes the vehicle body, wheels, navigation system, etc. of the AGV.

[0123] S205: Extract the contour and height information of the yard from the 3D point cloud model of the port; construct a yard model through 3D modeling software in combination with the types, sizes, and stacking methods of the goods in the yard;

[0124] In this embodiment, the yard model includes the ground of the yard, goods stacks, and possible obstacles, etc.

[0125] S206: Fuse the sensor data (such as positioning data, status data, motion data) with the quay crane model, AGV model, and yard model, and add corresponding sensor data attributes, such as position, speed, temperature, etc.;

[0126] S207: Assemble and integrate the quay crane model, AGV model, and yard model fused with sensor data to generate a port digital twin model.

[0127] II. Multi-agent reinforcement learning environment

[0128] Construct a multi-agent reinforcement learning environment through the following steps:

[0129] S301: Construct a quay crane agent, an AGV agent, and a yard agent;

[0130] Specifically:

[0131] The quay crane agent is responsible for controlling the operation of the quay crane to complete the cargo handling tasks, including the movement of the spreader and the loading and unloading of the cargo;

[0132] The AGV agent is responsible for controlling the movement and path planning of the AGV to complete the cargo transportation task;

[0133] The yard agent is responsible for monitoring and managing the stacking and retrieval of the yard cargo and optimizing the yard space utilization.

[0134] S302: Design the state space, action space, and reward function of the quay crane agent;

[0135] The state space is defined as the set of the position of the quay crane, the height of the spreader, the rotation angle of the spreader, and the target cargo position;

[0136] The action space is defined as the set of actions that the quay crane can execute, such as moving the spreader up and down, left and right, forward and backward, rotating the spreader, etc.;

[0137] The reward function is set according to the efficiency, accuracy, and safety of the quay crane in completing the task; for example, a positive reward is obtained for successfully loading and unloading the cargo, and a negative reward is obtained for collision or operation timeout;

[0138] The formula of the reward function is expressed as:

[0139] r aq =α 1 ·e aq -α 2 ·c aq -α 3 ·t aq ;

[0140] In the formula: e aq represents the speed of the quay crane in completing the task; c aq represents the collision of the quay crane; t aq represents the operation timeout of the quay crane; α 1 、α 2 、α 3 represent the weight coefficients; when calculating, it is necessary to standardize the parameters such as e aq 、c aq 、t aq to unify the dimension.

[0141] S303: Design the state space, action space, and reward function of the AGV agent;

[0142] The state space is defined as the set of the position, speed, direction, and target position of the AGV;

[0143] The action space is defined as the set of actions that the AGV can execute, such as accelerating, decelerating, turning, stopping, etc.;

[0144] The reward function is set according to the efficiency of the AGV reaching the target, the smoothness and safety of the path;

[0145] The formula of the reward function is expressed as:

[0146] r agv = β 1 ·(1 - t agv ) - β 2 ·p agv - β 3 ·c agv ;

[0147] In the formula: t agv represents the time required for the AGV to reach the target position; p agv represents the path smoothness; p agv represents that the AGV has a collision; β 1 、β 2 、β 3 represent the weight coefficients; when calculating, parameters such as t agv 、p agv 、p agv etc. need to be standardized to unify the dimension.

[0148] S304: Design the state space, action space and reward function of the yard intelligent agent;

[0149] The state space is defined as the set of the positions, types, quantities of goods in the yard and the free space in the yard;

[0150] The action space is defined as the set of actions that the yard can perform, such as stacking goods, retrieving goods, adjusting the positions of goods, etc.;

[0151] The reward function is set according to the yard space utilization rate, the goods retrieval efficiency and safety;

[0152] The formula of the reward function is expressed as:

[0153] r dc = γ 1 ·s dc - γ 2 ·r dc - γ 3 ·c dc ;

[0154] In the formula: s dc represents the yard space utilization efficiency; r dc represents the time required to retrieve goods; c dc represents that a collision occurs during the yard operation; γ 1 、γ 2 、γ 3 represent the weight coefficients; when calculating, s dc, r dc , c dc Standardize parameters such as etc. to unify the dimension.

[0155] S305: Use the port digital twin model as a multi-agent reinforcement learning environment, enabling the quay crane agent, AGV agent, and yard agent to interact and collaborate in the multi-agent reinforcement learning environment;

[0156] S306: Based on the state spaces, action spaces, and reward functions of the quay crane agent, AGV agent, and yard agent, construct the global state space, global action space, and global reward function of the multi-agent reinforcement learning environment, and simultaneously define the state transition function;

[0157] The global state space is the set of the state spaces of the quay crane agent, AGV agent, and yard agent;

[0158] The global action space is the set of the action spaces of the quay crane agent, AGV agent, and yard agent;

[0159] The global reward function is the sum of the reward functions of the quay crane agent, AGV agent, and yard agent;

[0160] The formula of the state transition function is expressed as:

[0161] S′ = f(S, A);

[0162] In the formula: S′ represents the new state after state transition; S represents the current state; A represents the current action of the agent; f is the function symbol.

[0163] III. Agent Training

[0164] Complete the training of each agent through the following steps:

[0165] S401: Initialize the port digital twin model in the multi-agent reinforcement learning environment, including the initial states of the quay crane model, AGV model, and yard model;

[0166] S402: Initialize the policy network and value network of the quay crane agent, AGV agent, and yard agent, and set the network structure, hyperparameters (such as learning rate, discount factor, etc.) and random seeds;

[0167] S403: Initialize an experience replay buffer to store the interaction experiences (state, action, reward, next state) of each agent in the environment; during training, every time an agent takes a step, its experience is stored in the buffer;

[0168] S404: Update the policies of each agent (Actor network) through the following steps:

[0169] S4041: For each agent, randomly sample a batch of experiences from the experience replay buffer;

[0170] S4042: Using the sampled experiences, calculate the action values under the current policy (evaluated by the Critic network);

[0171] S4043: Update the policy network parameters of each agent according to the ascending gradient of the action values to maximize the expected reward;

[0172] S405: Update the value function of each agent (Critic network) through the following steps:

[0173] S4051: For each agent, calculate the target values using the sampled experiences (based on the Bellman equation);

[0174] S4052: Use the mean squared error between the target values and the values predicted by the current value network as the loss function to update the value network parameters of each agent;

[0175] S406: Regularly (e.g., every certain number of steps) copy the parameters of the policy network and the value network to the target network of the corresponding agent to maintain target stability;

[0176] S407: Repeat steps S404 to S406 until the preset training steps or performance metrics are reached, and obtain the trained quay crane agents, AGV agents, and yard agents.

[0177] IV. Port Scheduling Strategy

[0178] Generate the port scheduling strategy through the following steps:

[0179] S501: Obtain the real-time task data and status data of the port;

[0180] The task data includes cargo handling task data, transportation task data, and special task requirements;

[0181] Cargo handling task data: includes cargo type, quantity, source location (such as the cabin on the ship), target location (such as the specific stacking position in the yard), etc.

[0182] Transportation task data: the cargo information that the AGV needs to transport, including cargo ID, starting point, ending point, etc.

[0183] Special task requirements: such as high-priority cargo, cargo that needs to be processed at a specific time, etc.

[0184] The status data includes quay crane status data, AGV status data, yard status data, and port environment status data;

[0185] Quayside crane status data: current position, spreader status (whether carrying goods), rotation angle, operation status (busy / idle), etc.

[0186] AGV status data: current position, speed, direction, cargo status, battery power, etc.

[0187] Yard status data: distribution of goods in each stack position, information on available stack positions, status of yard equipment (such as forklifts), etc.

[0188] Port environment status data: weather conditions, traffic conditions, safety alerts, etc.

[0189] S502: Map the status data (position, spreader status, etc.) of the quayside crane and the task data (loading and unloading tasks) to the input space of the quayside crane agent; the quayside crane agent outputs the operation actions of the quayside crane (such as moving direction, spreader operation, etc.) based on the input data and the trained policy network as the quayside crane scheduling strategy;

[0190] S503: Map the status data (position, speed, cargo status, etc.) of the AGV and the transportation task data to the input space of the AGV agent; the AGV agent outputs the driving path, speed adjustment, and steering of the AGV based on the input data and the policy network as the AGV scheduling strategy;

[0191] S504: Map the status data (goods distribution, available stack positions, etc.) of the yard and the yard operation tasks (stacking, retrieving goods) to the input space of the yard agent; the yard agent outputs the operation instructions of the yard equipment (such as goods stacking position, retrieval order, etc.) based on the input data and the policy network as the yard scheduling strategy;

[0192] S505: Integrate the quayside crane scheduling strategy, AGV scheduling strategy, and yard scheduling strategy (ensure the coordination and consistency between the strategies) to generate the port scheduling strategy.

[0193] V. Actual Scheduling

[0194] In this embodiment, the port scheduling strategy is decomposed into specific instructions and sent to the quayside crane equipment, AGV equipment, and yard equipment for execution respectively.

[0195] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit the technical solutions. Those of ordinary skill in the art should understand that any modifications or equivalent replacements made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions should be covered within the scope of the claims of the present invention.

Claims

1. A smart port scheduling and energy efficiency optimization method based on digital twins, characterized in that: include: S1: Acquire the 3D point cloud data and sensor data of the target port; S2: Build a port digital twin model including quay crane model, AGV model and yard model based on the 3D point cloud data and sensor data of the target port; S3: Build intelligent agents for the quay crane model, AGV model, and yard model in the port digital twin model, and build a multi-agent reinforcement learning environment; S4: Train each agent in a multi-agent reinforcement learning environment through a multi-agent reinforcement learning algorithm; S5: Map the real-time task data and status data of the port to the trained agents to simulate the scheduling of the target port and generate the port scheduling strategy; S6: Realize the operation scheduling of the quay crane equipment, AGV equipment and yard equipment of the target port based on the port scheduling strategy.

2. The method for smart port scheduling and energy efficiency optimization based on digital twins according to claim 1, characterized in that: In step S1, the three-dimensional point cloud data of the port includes ground point cloud data obtained by a ground laser scanning device on the ground where the port is located, and aerial point cloud data obtained by an unmanned aerial vehicle laser scanning device.

3. The method for smart port scheduling and energy efficiency optimization based on digital twins according to claim 2, characterized in that: In step S1, the sensor data of the port includes data collected by positioning sensors, state sensors and motion sensors.

4. The method for smart port scheduling and energy efficiency optimization based on digital twins according to claim 3 is characterized in that: In step S2, the port digital twin model is constructed through the following steps: S201: Preprocessing the three-dimensional point cloud data and sensor data including cleaning, denoising and registration; S202: Fusing the pre-processed ground point cloud data and the aerial point cloud data to obtain fused point cloud data; S202: Performing three-dimensional modeling based on the fused point cloud data by using a Poisson surface reconstruction algorithm to obtain a three-dimensional point cloud model of the port; S203: extracting characteristic points and contour lines of the quay crane equipment from the three-dimensional point cloud model of the port, and constructing a quay crane model based on the characteristic points and contour lines of the quay crane equipment by using three-dimensional modeling software; S204: constructing an AGV model based on the actual size and shape of the AGV device using a three-dimensional modeling software; S205: extracting the outline and height information of the yard from the three-dimensional point cloud model of the port; constructing a yard model by combining the types, sizes and stacking methods of the goods in the yard with the three-dimensional modeling software; S206: integrating the sensor data with the quay crane model, the AGV model and the yard model; S207: Assemble and integrate the quay crane model, AGV model and yard model that integrate sensor data to generate a digital twin model of the port.

5. The method for smart port scheduling and energy efficiency optimization based on digital twins according to claim 6 is characterized in that: In step S3, a multi-agent reinforcement learning environment is constructed through the following steps: S301: Construct the quay crane agent, AGV agent and yard agent; S302: Design the state space, action space and reward function of the quay crane agent; The state space is defined as the set of the position of the quay crane, the height of the spreader, the rotation angle of the spreader, and the target cargo position; The action space is defined as the set of actions that the quay crane can perform; The formula of the reward function is expressed as: r aq =α1·e aq -α2·c aq -α3·t aq ; Where: e aq Indicates the speed at which the quay crane completes its task; c aq Indicates that the quay crane has collided; t aq Indicates that the quay crane operation is overtime; α1, α2, α3 represent weight coefficients; S303: Design the state space, action space and reward function of the AGV agent; The state space is defined as the set of the AGV’s position, velocity, direction, and target position; The action space is defined as the set of actions that the AGV can perform; The formula of the reward function is expressed as: r agv =β1·(1-t agv )-β2·p agv -β3·c agv ; Where: t agv Indicates the time required for AGV to reach the target location; p agv represents path smoothness; p agv Indicates that the AGV has collided; β1, β2, and β3 represent weight coefficients; S304: Design the state space, action space and reward function of the yard agent; The state space is defined as the set of the location, type, quantity of goods in the yard and the free space in the yard; The action space is defined as the set of actions that the yard can perform; The formula of the reward function is expressed as: r dc =γ1·s dc -γ2·r dc -γ3·c dc ; Where: s dc Indicates the utilization efficiency of the yard space; r dc Indicates the time required to retrieve the goods; c dc Indicates that a collision occurs during yard operation; γ1, γ2, and γ3 represent weight coefficients; S305: Use the port digital twin model as a multi-agent reinforcement learning environment, so that the quay crane agent, AGV agent and yard agent can interact and collaborate in the multi-agent reinforcement learning environment; S306: construct the global state space, global action space and global reward function of the multi-agent reinforcement learning environment based on the state space, action space and reward function of the quay crane agent, AGV agent and yard agent, and define the state transfer function; The global state space is the collection of state spaces of the quay crane agent, AGV agent, and yard agent; The global action space is the collection of action spaces of the quay crane agent, AGV agent, and yard agent; The global reward function is the sum of the reward functions of the quay crane agent, AGV agent, and yard agent; The formula of the state transfer function is expressed as: S′=f(S,A); In the formula: S′ represents the new state after state transfer; S represents the current state; A represents the current action of the agent; f is the function symbol.

6. The method for smart port scheduling and energy efficiency optimization based on digital twins according to claim 5, characterized in that: In step S301: The quay crane agent is responsible for controlling the operation of the quay crane to complete cargo loading and unloading tasks; The AGV agent is responsible for controlling the movement and path planning of the AGV to complete the task of transporting goods; The yard agent is responsible for monitoring and managing the stacking and retrieval of goods in the yard.

7. The method for smart port scheduling and energy efficiency optimization based on digital twins according to claim 5, characterized in that: In step S4, the training of each agent is completed through the following steps: S401: Initialize the port digital twin model in a multi-agent reinforcement learning environment, including the initial states of the quay crane model, AGV model, and yard model; S402: Initialize the strategy network and value network of the quay crane agent, the AGV agent, and the yard agent; S403: Initialize an experience playback buffer for storing the interaction experience of each agent in the environment; during the training process, each time the agent executes a step, its experience is stored in the buffer; S404: Update the strategies of each agent through the following steps: S4041: For each agent, randomly sample a batch of experiences from the experience replay buffer; S4042: Using the sampled experience, calculate the action value under the current strategy; S4043: Update the policy network parameters of each agent according to the action value gradient ascent to maximize the expected reward; S405: Update the value function of each agent through the following steps: S4051: For each agent, use the sampled experience to calculate the target value; S4052: Use the mean square error between the target value and the value predicted by the current value network as the loss function to update the value network parameters of each agent; S406: Periodically copy the parameters of the policy network and the value network to the target network of the corresponding agent; S407: Repeat steps S404 to S406 until a preset number of training steps or performance indicators are reached, thereby obtaining trained quay crane agents, AGV agents, and yard agents.

8. The method for smart port scheduling and energy efficiency optimization based on digital twins according to claim 7, characterized in that: In step S501, the task data includes cargo loading and unloading task data, transportation task data, yard operation task data and special task requirements; The status data includes quay crane status data, AGV status data, yard status data and port environment status data.

9. The method for smart port scheduling and energy efficiency optimization based on digital twins according to claim 8, characterized in that: In step S5, the port scheduling strategy is generated through the following steps: S501: Acquire real-time task data and status data of the port; S502: Mapping the quay crane status data and cargo loading and unloading task data to the input space of the quay crane agent; the quay crane agent outputs the quay crane operation action as the quay crane scheduling strategy according to the input data and the strategy network; S503: Mapping the AGV status data and the transportation task data to the input space of the AGV agent; the AGV agent outputs the driving path, speed adjustment and steering information of the AGV device as the AGV scheduling strategy based on the input data and the strategy network; S504: Mapping the yard status data and the yard operation task data to the input space of the yard agent; the yard agent outputs the operation instructions of the yard equipment as the yard scheduling strategy according to the input data and the strategy network; S505: Integrate the quay crane scheduling strategy, AGV scheduling strategy and yard scheduling strategy to generate a port scheduling strategy.

10. The method for smart port scheduling and energy efficiency optimization based on digital twins according to claim 1, characterized in that: In step S6, the port dispatching strategy is decomposed into specific instructions and sent to the quay crane equipment, AGV equipment and yard equipment for execution.

Images