A Potential Game Theory-Driven Deep Reinforcement Learning-Based Automated Container Terminal AGV Group Charging Decision Method

Abstract

This invention discloses a charging decision-making method for AGV groups in automated container terminals based on potential game deep reinforcement learning. It constructs an energy balance system model for the AGV group in the automated terminal, establishes a potential game model for AGV group energy balance, maps the charging decision behavior of each AGV to a game strategy, and constructs a potential function. A reward function is designed, and a multi-agent deep reinforcement learning framework is built based on the reward function. Each AGV is treated as an independent decision-making agent, and joint training is performed during the training phase through a centralized value evaluation and distributed policy execution mechanism. During training, the AGVs continuously interact with the simulation environment, and the policy network and value evaluation network are iteratively updated. After training, the policy networks of each agent are deployed on the AGVs, which autonomously decide on charging based on local observation information, achieving optimized charging and global energy balance for AGV groups in multi-layout terminals, significantly improving charging resource utilization and system robustness.

Description

Technical Field

[0001] This invention belongs to the field of intelligent scheduling technology for automated terminals, specifically involving a charging decision-making method for AGV groups in automated container terminals driven by potential game deep reinforcement learning. Background Technology

[0002] With the continuous improvement of port automation and intelligence, automated container terminals have become an important direction for modern port construction and development. Automated terminals, through the introduction of highly integrated information systems and intelligent operating equipment, realize the coordinated operation of loading, unloading, transportation, and storage, and have significant advantages in improving operational efficiency, reducing labor costs, enhancing operational safety, and promoting the green and low-carbon development of ports.

[0003] Automated Guided Vehicles (AGVs), as key equipment in automated container terminals undertaking horizontal transport tasks, are primarily responsible for the high-frequency transfer of containers between quay cranes and yard cranes. Battery-powered AGVs are gradually replacing traditional fuel-powered equipment, becoming the mainstream configuration for automated terminals. Compared to fuel-powered vehicles, battery-powered AGVs have advantages such as lower energy consumption, lower emissions, and lower noise, helping to reduce the environmental impact of port operations.

[0004] However, limited by battery capacity and charging / discharging characteristics, the range of automated guided vehicles (AGVs) and charging scheduling methods have become crucial factors restricting the overall operational efficiency of automated terminals. During peak ship berthing periods, AGVs need to continuously perform high-intensity transfer tasks, leading to rapid power consumption. If charging decisions are not made appropriately, multiple vehicles may rush to charging stations for refueling within a short period, resulting in queues. This phenomenon can create an "energy bottleneck" in both time and space: on the one hand, increased queuing time during peak charging periods directly affects the connection efficiency between quay cranes and yard cranes, extending the time ships spend in port; on the other hand, during off-peak periods, some charging stations have low utilization rates, with charging facilities remaining idle for extended periods, resulting in insufficient resource allocation efficiency.

[0005] The charging decision-making methods commonly used in existing automated terminals are mostly rule-based strategies based on fixed thresholds or proximity principles. For example, when a vehicle's battery level falls below a preset threshold, it will proceed to the nearest charging station. While these methods are simple to implement, they primarily rely on the local state of a single vehicle for decision-making, lacking a holistic consideration of the behavior of the automated guided vehicles (AGVs) as a group and the competition for charging resources. This can easily lead to a negative cumulative effect of individual decisions at the group level, resulting in the problem of "individual rationality but overall efficiency decline," making it difficult to meet the comprehensive requirements of energy balance and system efficiency for large-scale automated terminals.

[0006] To improve the intelligence level of charging scheduling, some existing technologies attempt to introduce reinforcement learning methods to optimize vehicle charging behavior. For example, existing technology CN119669696A discloses a charging control method for electric vehicle charging stations based on deep reinforcement learning. By acquiring charging station information and electric vehicle information, and aiming to meet the charging needs of electric vehicles and maximize the profits of charging stations, a reward function is constructed. The charging coordination control problem within the charging station is modeled as a Markov random game model, and a centralized Critic multi-agent reinforcement learning algorithm is used to solve the charging strategy.

[0007] However, the aforementioned existing technologies primarily target electric vehicle charging scheduling scenarios within urban public charging stations or enclosed charging stations. Their research focuses on charging equipment or chargers as the decision-making entities, emphasizing charging station revenue or local charging coordination effects. They fail to adequately consider the characteristics of automated guided vehicles (AGVs) in automated terminal scenarios, such as high mobility, high task coupling, and a strong correlation between charging behavior and logistics scheduling during transportation tasks. Furthermore, these methods typically emphasize coordination and control within the charging station, lacking a unified model of the overall energy distribution of the AGV group, making it difficult to achieve energy balance at the group level.

[0008] Furthermore, different automated terminals exhibit significant differences in spatial layout and operational organization. Existing terminal layouts mainly include vertical, U-shaped, and horizontal layouts. Under different layout conditions, the distance between quay cranes and the yard, the travel paths of automated guided vehicles (AGVs), the spatial distribution of charging stations, and their service range all vary considerably, resulting in complex coupling relationships between charging behavior, traffic flow, and operational scheduling. In vertical layouts, charging stations are mostly concentrated in the yard area, easily leading to localized traffic congestion during peak operating periods. In U-shaped layouts, charging stations are dispersed inside and outside the circular passageway, requiring charging decisions to consider travel path conflicts. In horizontal layouts, although vehicles can recharge nearby during operations, the number of charging facilities and the grid load increase significantly with the increase in quay length. Consequently, the same charging strategy cannot achieve consistent energy balance under different layout conditions.

[0009] In summary, existing technologies for charging decisions of automated guided vehicles (AGVs) at automated terminals generally suffer from the following shortcomings: First, they lack a unified decision-making mechanism for coordinating the energy distribution of AGVs at the group level; second, they struggle to characterize the game-theoretic relationships among multiple vehicles vying for limited charging resources; and third, they lack adaptability and scalability for charging strategies under different terminal layout conditions. Therefore, there is an urgent need for a charging decision-making method that can combine the layout characteristics of automated terminals, achieve energy balance of AGVs at the group level, and possess adaptive learning capabilities, in order to improve the operational efficiency and energy utilization level of automated terminals under complex operating conditions. Summary of the Invention

[0010] The purpose of this invention is to overcome the shortcomings of the existing technology and provide an automated container terminal AGV group charging decision-making method based on potential game deep reinforcement learning.

[0011] The objective of this invention can be achieved through the following technical solutions: This invention provides a charging decision-making method for AGV groups in automated container terminals based on potential game deep reinforcement learning, comprising the following steps: Construct an energy balancing system model for an automated terminal AGV group; Based on the system model, an AGV group energy balance potential game model is established, which maps the charging decision behavior of each automated guided vehicle to a game strategy, and constructs a potential function consistent with the overall energy balance goal of the AGV group. Based on the aforementioned potential game model, a reward function with potential function changes as its core is designed; Based on the reward function, a potential game reinforcement learning algorithm framework is constructed, treating each automated guided vehicle as an independent decision-making agent. Based on local observation information, it outputs whether to charge and the corresponding charging control decision. In the training phase, a centralized value evaluation and distributed policy execution mechanism are introduced to jointly train the charging strategies of each automated guided vehicle. During the joint training process, the policy network and value evaluation network of each agent are iteratively updated through continuous interaction between each automated guided vehicle and the automated terminal simulation environment. After training, the policy networks of each agent are deployed to each automated guided vehicle (AGV). In actual operation, the AGVs autonomously make charging decisions based on their local states, thereby achieving distributed charging control and global energy balance for the AGV group under different dock layout conditions.

[0012] Furthermore, the construction of the energy balancing system model for the automated terminal AGV group specifically includes: An aggregated model is used to model the automated terminal operation system, defining an automated guided vehicle (AGV) set, a charging station set, a system time set, and a system operation state set. The AGV set represents multiple AGVs participating in charging decisions, the charging station set represents the charging facilities in the system that can charge AGVs, and the system operation state set describes the overall operation state of the automated terminal at any given time. Based on the aforementioned aggregated modeling, an energy balancing system model for an automated terminal AGV group is constructed. The remaining power status, operating position status, and task execution status of each automated guided vehicle at any given time, as well as the service occupancy status, queuing status, and spatial position status of each charging station at the corresponding time, are uniformly mapped to the global status of the facility, representing the comprehensive operating status of the system at that time. For the global state of the facility, a state reward function is defined to characterize the quality of system operation, expressed as: in, Indicates the global status of the facility The corresponding state reward value; This indicates the overall power level of the AGV group. This indicates the degree of dispersion in the power consumption of the AGV group. This indicates penalties for the use of charging station resources. , and These represent the weight coefficients of the corresponding items; Based on the aforementioned state-reward function, a charging strategy for the AGV group in any given group is defined. The individual benefit function for each automated guided vehicle is expressed as follows: in, Indicates the first Group charging strategy for automated guided vehicles Individual earnings value under the following conditions This refers to a group charging strategy comprised of the charging strategies of each automated guided vehicle (AGV). Indicates the first Charging strategy for autonomous vehicles Indicates the first A set of optional charging strategies for each automated guided vehicle; This indicates the charging strategy. The corresponding global state set of facilities.

[0013] Furthermore, the overall power level of the AGV group , is represented as: in, Indicates the first Automated guided vehicles at time The remaining battery power, Indicates the number of automated guided vehicles (AGVs) participating in the charging decision-making process; The dispersion of the AGV group's power consumption , is represented as: The charging station resource utilization penalty item , is represented as: in, This indicates the penalty item corresponding to the charging station being idle. This indicates the penalty item corresponding to the queuing status at the charging station. These represent the weight coefficients of the charging station idle penalty item and the queuing penalty item, respectively.

[0014] Furthermore, the penalty item corresponding to the idle state of the charging station , is represented as: in, Indicates the first A charging station at a time The service occupancy status is 1, which indicates that the charging station is in service, and 0 indicates that the charging station is idle. Indicates the number of charging stations; This represents the total statistical time of system operation. The penalty item corresponding to the queuing status at the charging station , is represented as: in, Indicates the first A charging station at a time The queue length, i.e., the charging station at any given time. Number of automated guided vehicles waiting to be charged.

[0015] Furthermore, based on the aforementioned system model, an AGV group energy balance potential game model is established, mapping the charging decision behavior of each automated guided vehicle to a game strategy, and constructing a potential function consistent with the overall energy balance objective of the AGV group, specifically including: The charging strategy of each automated guided vehicle is represented as an AGV group game strategy. ,in, Indicates the first Charging strategy for autonomous vehicles Indicates except the first Charging strategy for all automated guided vehicles except for the 1000 Automated Guided Vehicles; Based on the individual payoff functions of each automated guided vehicle (AGV) in the system model, an energy equilibrium potential game model for the AGV swarm is constructed, specifically including: In the aforementioned potential game model, a strategy update rule is set: when a strategy exists... Make Then the group charging strategy will be changed from Updated to ;in, This indicates a new group charging strategy; Indicates the first A new charging strategy for automated guided vehicles; Indicates the first Automated guided vehicles in the original group strategy Individual profit value; Indicates the first Automated guided vehicles in the updated swarm strategy Individual profit value; In the aforementioned potential game model, the Nash equilibrium condition is defined if and only if At that time, the set of strategies For the Nash equilibrium solution, where, Indicates the first A set of optional charging strategies for each automated guided vehicle.

[0016] Furthermore, the potential function is expressed as: in, Representing group policy The corresponding potential function value, This represents the total statistical time of system operation. Represents the global state set of the facility. Indicates at time Group strategy The corresponding state is in the global facility status. The number of AGVs, Indicates the global status of the facility The profit value.

[0017] Furthermore, the reward function is defined by the following formula: in, Indicates the first Automated guided vehicles at time The reinforcement learning reward value, Indicates time Downstream charging strategy The corresponding potential function value, Indicates time Downstream charging strategy The corresponding potential function value.

[0018] Furthermore, the aforementioned framework for constructing a potential game-theoretic reinforcement learning algorithm based on the reward function treats each automated guided vehicle (AGV) as an independent decision-making agent. Based on local observation information, each AAV outputs a charging decision and the corresponding charging control decision. During the training phase, a centralized value evaluation and distributed policy execution mechanism is introduced to jointly train the charging strategies of each AAV. Specifically, this includes: The AGV swarm is modeled as a multi-agent system, and the local observations of each automated guided vehicle are represented as follows: ,in, For the first The current battery level of the automated guided vehicle. For the first The current location of the automated guided vehicle. For the first The current mission status of the automated guided vehicle. For the first The target point of the automated guided vehicle. For the first Information on nearby charging stations for each automated guided vehicle includes location, queue length, and occupancy status. For the first Information about the surrounding AGVs of an automated guided vehicle, including their relative positions and task levels; At the same time, define the global state. ,in, This indicates the overall power distribution range of the AGV group. This represents the set of task levels for the AGV group. This is a set of AGV group position states. The first The status, queue length, and spatial location of each charging station. This refers to the set of automated guided vehicles (AGVs) that participate in the charging decision-making process. This represents the set of charging stations in the system; The actions of each automated guided vehicle are represented as a hybrid action space. Among them, discrete actions This indicates either to continue the task or to initiate a charging process. Indicates that charging has been initiated. Indicates continuation of the task; continuous actions. Used to select the initial charging threshold. and termination charging threshold ; The Actor-Critic framework is constructed based on a multi-agent deep deterministic policy gradient-potential game algorithm, with local observations input into the Actor network. Output action The Actor network adopts parameter sharing, meaning all automated guided vehicles (AGVs) share the same set of Actor network parameters; the Critic network inputs the AGV group charging strategy. s and joint actions Output potential function value This is used to evaluate the contribution of each AGV's actions to the group's energy balance. This indicates the number of automated guided vehicles (AGVs) involved in the charging decision-making process.

[0019] Furthermore, the loss function of the Critic network is expressed as: in, Indicates the first Parameters of the Critic network for automated guided vehicles. This represents the expectation for the experience replay sample. Indicates the parameters of the Critic network Below, group charging strategy and joint actions The corresponding potential function value, Indicates the first The target value for each automated guided vehicle is calculated using the following formula: in, Indicates the first Automated guided vehicles at time The reward value, This is a discount factor used to reflect the importance of future rewards. This represents the potential function value output by the Critic target network. This represents the action output by the Actor target network. Indicates the first Parameters of the Critic target network for an automated guided vehicle. This represents the updated set of group charging strategies; The Actor network parameters are updated via gradient ascent: in, Indicates the first Parameters of the Actor network for automated guided vehicles. This represents the objective function for the charging strategy performance. Indicates the first The gradient of the movement of an automated guided vehicle. This represents the potential function value output by the Critic network. This indicates that the Actor network is based on local observations. The output action; Indicates the first The gradient of the parameters of the Actor network for an automated guided vehicle.

[0020] Furthermore, the joint training specifically includes: During the training phase, the automated terminal simulation environment is used as a multi-agent interaction environment. The Actor network and Critic network of each automated guided vehicle are initialized, and an experience playback storage module is configured for each automated guided vehicle to store training sample data. During the interaction phase, for any given training moment, each automated guided vehicle (AGV) bases its training on the acquired local observations. Each actor network outputs its corresponding blended action. and in the mixed action By introducing exploratory noise, the combined actions of each automated guided vehicle (AGV) are applied to the automated terminal simulation environment to obtain an updated group charging strategy. and environmental status feedback; During the interaction phase, the reward value obtained by each automated guided vehicle at the current moment is calculated based on the reward function. and the mixed movements of all automated guided vehicles The current group charging strategy constitutes Local observation Mixed actions Reward Value and the updated group charging strategy Composition of empirical samples Stored in the experience replay pool; During the update phase, a batch of experience samples are randomly sampled from the experience replay pool. Based on the loss function and parameter update method, the Critic network parameters and Actor network parameters of each automated guided vehicle are updated. The Critic network adopts a centralized value evaluation method, and the Actor network adopts a distributed strategy update method.

[0021] Compared with the prior art, the present invention has the following advantages: (1) In the prior art, the charging decisions of AGVs in automated terminals are mostly based on fixed power thresholds or the principle of proximity, and only make decisions based on the local state of a single vehicle. This fails to depict the competitive relationship among multiple AGVs around limited charging resources, which easily leads to concentrated charging during peak hours and idle resources during off-peak hours, resulting in energy distribution imbalance and overall operating efficiency decline at the group level. To address this problem, this invention introduces a potential game modeling method based on the construction of an energy balance system model for an automated terminal AGV group. This method maps the charging decision behavior of each AGV to a unified game strategy, and constructs a potential function consistent with the overall energy balance goal of the AGV group. This allows individual strategy improvements to strictly correspond to the improvement of the group goal, thereby achieving comprehensive optimization of the overall power level, power distribution balance, and charging station resource utilization of the AGV group. This effectively avoids the phenomenon of individual rationality and overall deterioration, and reduces the overall operating cost of the terminal.

[0022] (2) Existing multi-vehicle charging scheduling methods based on reinforcement learning or heuristic optimization often lack clear game structure constraints, making it difficult to guarantee algorithm convergence and stability. They are prone to policy oscillations or long-term non-convergence, affecting the reliability of actual deployment. To address this problem, this invention constructs an AGV group energy equilibrium potential game model, clarifying the consistency relationship between the potential function and the individual payoff function of each AGV. This ensures that any policy improvement of any AGV will cause a monotonically increasing potential function, theoretically guaranteeing the existence of the game's Nash equilibrium. This ensures that the charging decision algorithm designed based on this model can converge to a stable policy solution, significantly improving the stability and predictability of system operation.

[0023] (3) Existing centralized charging control methods rely on a central controller to make unified decisions on the status and actions of all vehicles. As the number of AGVs increases, the communication overhead and computational complexity rise rapidly, and the system is highly dependent on the central node, resulting in poor robustness. To address this problem, this invention constructs a multi-agent deep reinforcement learning framework based on a potential game model, treating each AGV as an independent decision-making agent. It adopts the MADDPG algorithm structure, which combines centralized value evaluation with distributed policy execution. During the training phase, global information is used to evaluate the value function, while during the execution phase, only the local observation information of the AGV itself is relied upon for autonomous decision-making. This enables collaborative charging decision-making of the AGV group without relying on a central controller, reducing the system communication burden and improving the system's distributed autonomy and operational robustness.

[0024] (4) Existing technologies mostly model single or simplified scenarios, which are difficult to adapt to the actual situation where there are significant differences in AGV driving paths, charging station distribution and traffic topology under different automated terminal layouts, resulting in poor portability of charging strategies between different terminals. To address this problem, this invention uses a set-based approach to uniformly describe the AGV group state and charging station state during the system modeling stage. In the reinforcement learning framework, a global state representation is designed that includes the AGV group energy state, spatial location, task state and charging station occupancy and queuing information. This enables the constructed charging decision model to naturally adapt to different automated terminal structures such as vertical layout, U-shaped layout and horizontal layout, and realizes flexible application of the same algorithm framework with different layout parameter configurations, significantly improving the portability and engineering applicability of the method.

[0025] (5) Existing charging scheduling methods typically only aim to meet vehicle charging needs or shorten individual vehicle waiting times, without systematically constraining the efficiency of charging station resource utilization. This can easily lead to problems such as long-term idle charging facilities or severe local congestion. To address this issue, this invention explicitly introduces charging station idle penalty and queuing penalty terms into the system model, incorporating charging station utilization efficiency into the design of the potential and reward functions. This allows AGVs to comprehensively weigh their own power status and the charging station load when making charging decisions, thereby guiding the AGV group to achieve balanced use of charging resources in both time and space dimensions, avoiding resource waste and service congestion, and improving the overall utilization rate of charging facilities.

[0026] (6) As the scale of port operations changes, the number of AGVs may be dynamically adjusted. Existing methods require redesigning controllers or retraining models, resulting in poor scalability. To address this issue, this invention employs an Actor network parameter sharing mechanism within a multi-agent reinforcement learning framework. This allows all AGVs to share the same set of policy network parameters, reducing the model parameter size and training complexity. Furthermore, it supports flexible increases or decreases in the number of AGVs, enabling plug-and-play deployment of AGV groups of different sizes. This enhances the scalability and long-term applicability of the method in actual port operations. Attached Figure Description

[0027] Figure 1 This is a schematic diagram of the AGV group charging decision method according to an embodiment of the present invention; Figure 2 These are schematic diagrams of three typical layouts of automated terminals according to embodiments of the present invention; Figure 3 This is a flowchart of the training process of the MADDPG-PGT algorithm according to an embodiment of the present invention. Detailed Implementation

[0028] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0029] Example 1: This embodiment provides a potential game-based deep reinforcement learning-driven AGV group charging decision-making method. By transforming the AGV group energy balance game into a potential game model, and designing a potential game-based reinforcement learning algorithm based on Multi-Agent Deep Deterministic Policy Gradient (MADDPG), it addresses the conflict between individual rationality and group energy efficiency during AGV charging at automated terminals, achieving an energy balance state where the AGV group has a uniform power level and charging station resources are utilized appropriately. Figure 1As shown, it includes the following steps: S1, System Model Construction Module: Constructs an AGV group energy balance system model; Define the relevant set: Let the set of AGV numbers be... Collection of charging station numbers AGV group state set Charging station status set global state set of facilities Time set , charging strategy set .

[0030] global status of facilities Benefits Defined as: in, Indicates the global status of the facility The corresponding state reward value; This indicates the overall power level of the AGV group. This indicates the degree of dispersion in the power consumption of the AGV group. This indicates penalties for the use of charging station resources. , and These represent the weighting coefficients for the corresponding items; they are based on the AGV group's power level. Based on the baseline, when the power distribution range Increase the utilization of charging station resources When things are unreasonable, the returns will decrease, that is... , The larger, The smaller. , and The weights are, in order, the AGV group's power level, power distribution, and charging station resource utilization. ,and , , .

[0031] AGV group charging strategy s The following returns are: in, Indicates the first Group charging strategy for automated guided vehicles Individual earnings value under the following conditions This refers to a group charging strategy comprised of the charging strategies of each automated guided vehicle (AGV). Indicates the first Charging strategy for autonomous vehicles Indicates the first A set of optional charging strategies for each automated guided vehicle; Representation and charging strategy The corresponding global state set of facilities.

[0032] Overall power level of AGV group , is represented as: in, Indicates the first Automated guided vehicles at time The remaining battery power, Indicates the number of automated guided vehicles (AGVs) participating in the charging decision-making process; AGV group power dispersion , is represented as: Penalties for the use of charging station resources , is represented as: in, This indicates the penalty item corresponding to the charging station being idle. This indicates the penalty item corresponding to the queuing status at the charging station. These represent the weight coefficients for the charging station idle penalty and the queuing penalty, respectively.

[0033] S2, Potential Game Model Building Module: Construct an energy equilibrium potential game model for the AGV group and prove the existence of Nash equilibrium; Definition 1 (Policy Update) Better Policy Update: If a policy exists... AGV i Benefits The group charging strategy is then determined by Updated to Optimal strategy update: AGV i Choose the strategy with the greatest increase in returns. .

[0034] Definition 2 (Nash Equilibrium): In an energy equilibrium game of AGV swarms, if and only if Time, Strategy This is the Nash equilibrium solution, and the AGV cannot further increase its revenue by changing the charging strategy.

[0035] Definition 3 (Potential Game): A game is called a potential game if and only if there exists a potential function. ,satisfy: Construct the potential function, that is, the potential function of the AGV group energy equilibrium game model: in, Under the group charging strategy at time t, the facility is in a global state. The number of AGVs. (Based on the overall facility status.) Belonging AGV i Global state of charging strategy The situation will be used to collect facility status data. Divided into four mutually exclusive subsets: Substituting the potential functions of the four facility state sets into the potential game yields: Therefore, the energy equilibrium game model of the AGV group is a potential game model.

[0036] S3, Reward Function Design Module: Design a reward function based on the change of potential function; Each AGV in an automated terminal is considered an autonomous decision-making agent. These agents interact within a shared environment in a multi-agent system, aiming to maintain overall system energy balance while completing transportation tasks. Cooperation between agents is achieved through a system model and a potential game model; that is, individual behavior is guided by shared reward signals, ensuring that agents not only focus on their own gains but also serve the global system goal. Each agent maximizing its accumulated reward is equivalent to optimizing the global energy balance objective, namely maintaining a uniform power level, moderate charging station utilization, and reducing queuing and idle time.

[0037] in, Indicates the first Automated guided vehicles at time The reinforcement learning reward value, Group charging strategy Next action Afterwards, the system state changed, and the group policy was updated to... The reward they receive is equal to their contribution to the change in the potential function. Indicates time Downstream charging strategy The corresponding potential function value, Indicates time Downstream charging strategy The corresponding potential function value. The change in penalty value for inappropriate use of the charging station before and after the action; The dynamic equation for the inappropriate utilization penalty function driven by changes in charging station idle time and queue length.

[0038] S4, MADDPG-PGT Algorithm Design Module: Design a multi-agent deep deterministic policy gradient-potential game algorithm; AGV swarms belong to multi-agent systems, and charging strategies directly affect the status of the AGVs themselves, other AGVs, and the charging station. AGV task levels are classified into levels I-VI based on "having / not having a task, having / not having power, and whether the remaining power is sufficient to complete the current task." e The remaining power of the AGV. The initial charging threshold, The threshold for terminating charging.

[0039] Based on the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm as its core framework, and incorporating Potential Game Theory (PGT), a MADDPG-PGT algorithm is proposed. To find the Nash equilibrium, distributed training is employed, and global state and local observations are distinguished. Global state ,in, This indicates the overall power distribution range of the AGV group. This represents the set of task levels for the AGV group. This is a set of AGV group position states. The first The status, queue length, and spatial location of each charging station. This refers to the set of automated guided vehicles (AGVs) that participate in the charging decision-making process. This represents the set of charging stations in the system; Local observation . , , , AGVs in order i Current battery level, current location, current task status, and target point; , AGV iInformation on nearby charging stations (location, queue length, status) and AGV information (relative location, task level).

[0040] Hybrid motion space Among them, discrete actions To continue the task, 1 initiates charging; continuous action. When selecting charging, output the charging threshold. and termination charging threshold , range [0,1].

[0041] In MADDPG-PGT, the Actor network is responsible for decision-making, taking local observations as input. Output action The Critic network is responsible for evaluating and inputting the AGV group charging strategy. and joint actions Output Value. Due to the isomorphism of AGVs, the Actor network uses parameter sharing to accelerate convergence and ensure policy consistency. Critic network loss function. ,in For AGV i Critic network parameters W For the expected value of the experience replay sample, For AGV i The target value, For AGV i In the Critic network, the action value potential function is used. The objective value is... ,in It's a discount factor that reflects the importance of future rewards. and AGV i The Critic and Actor target networks, For AGV i The Critic target network parameters. The Actor network is updated to... ,in Let U be the gradient of the Actor network parameters, and U be the objective function for the charging strategy performance. For AGV i The action gradient, Output network for Actor. Target network soft update to... ,in For the current network parameters, For the target network parameters, This is the soft update coefficient.

[0042] The distributed training process is as follows: (1) Initialization: Randomly initialize the Actor and Critic networks and their target network; empty the experience replay pool; (2) Interaction phase: Each agent, based on observations... Select Action And incorporate exploration noise. Next strategy based on environmental feedback. And calculate the reward Empirical tuples (3) Update phase: Sample batch experience from the replay pool and update the Actor and Critic networks. Use the target network soft update to adjust the target network parameters.

[0043] S5, Energy Balance Decision Execution Module: Executes AGV group charging decisions based on the trained strategy network to achieve energy balance.

[0044] The trained Actor network is deployed to each AGV. The AGV makes real-time decisions on whether to charge and the charging threshold based on local observations, thereby achieving distributed energy balance control of the AGV group.

[0045] Example 2: To better understand the application scenarios of this invention, such as Figure 2 As shown, three typical layouts of automated terminals are illustrated. Figure 2 (a) is a vertical layout, with the quay cranes arranged vertically to the yard, resulting in a short AGV travel path and charging stations concentrated at the rear of the yard. Figure 2 (b) is a U-shaped layout, with the quay crane and storage yard arranged along a semi-enclosed corridor, the AGV travel path is circular, and the charging stations are scattered between the inner and outer rails. Figure 2 (c) is a horizontal layout, where the quay crane and storage yard are in a parallel strip, the AGV travels in a straight, reciprocating path, and charging stations are equidistantly arranged along both sides of the travel strip. In all three layouts, the method of this invention can achieve energy balance by adjusting global state parameters.

[0046] like Figure 3 The diagram shown illustrates the training flowchart of the MADDPG-PGT algorithm of this invention. First, the Actor and Critic networks and their target network are initialized, and the experience replay pool is cleared. During the interaction phase, each AGV agent selects an action based on local observations and adds exploration noise. The environment provides feedback on the next state and reward, and the experience tuples are stored in the replay pool. During the update phase, a batch of experience is sampled from the replay pool, the Actor and Critic networks are updated, and the target network parameters are softly updated. The interaction and update phases are repeated until convergence.

[0047] Table 1 illustrates the task level classification of the AGV according to the present invention. Based on whether the AGV has a task and whether the remaining power is sufficient, the AGV is divided into six task levels, I-VI, with different charging strategies corresponding to different levels: Levels I-III prioritize charging to full capacity; Levels IV-VI charge flexibly while ensuring task completion.

[0048] Table 1 AGV Task Level Classification Table The following is a detailed description of the specific implementation methods of each module of the present invention: S1, System Model Construction Module: To construct an energy balance system model for an AGV group, the following assumptions are set to simplify model analysis and focus on the core optimization problem: Assumption 1: The current battery level, location, and task status (with / without task, task target point) of each AGV can be obtained in real time and shared through communication.

[0049] Assumption 2: The busy status (idle / busy / faulty), queue length, and location of each charging station can be obtained in real time and broadcast through the port communication network.

[0050] Assumption 3: All AGVs are isomorphic, meaning they have the same battery capacity, charging power, and driving energy consumption characteristics, and can use the same Actor network parameters.

[0051] Assumption 4: The AGV's travel path is predetermined by the dock path planning system. This invention only optimizes the charging decision and does not optimize the travel path.

[0052] Assumption 5: The charging power of the charging station is constant, and the charging time of the AGV is linearly related to the charging amount.

[0053] Assumption 6: The AGV can make real-time decisions on whether to go to the charging station during the transportation task, allowing the task to be interrupted for charging (except for emergency tasks).

[0054] Based on the above assumptions, a system model as described in the invention is constructed, defining global state, revenue function, power level, power distribution range, charging station penalty function, etc.

[0055] S2, Potential Game Model Building Module: Based on the constructed system model, and in accordance with Definitions 1-3 and the potential function construction method described in the invention, an AGV group energy equilibrium potential game model is established.

[0056] In practice, the AGV group charging strategy is first initialized as a random strategy. Then, the strategy update is performed iteratively: each AGV checks in turn whether there is a better strategy, and if so, updates to the better or optimal strategy. Due to the finite improvement property of potential games, this process will converge to Nash equilibrium within a finite number of steps.

[0057] S3, Reward Function Design Module: The reward function is designed based on the change of potential function, so that each AGV maximizes its own cumulative reward, which is equivalent to optimizing the global energy balance objective.

[0058] In practice, at each decision-making moment Calculate the current global state potential function AGVi's group charging strategy Next action Afterwards, the system state changed, and the group policy was updated to... ,award ,like This indicates that the action improved the global energy balance; if This indicates that the action worsened the global energy balance.

[0059] S4, MADDPG-PGT algorithm design module: like Figure 3 As shown, the specific implementation steps of the MADDPG-PGT algorithm are as follows: S41. Network Initialization: Randomly initialize the Actor network, Critic network, and their target network. S42. Hyperparameter settings: Set population size (number of AGVs), maximum number of iteration rounds, maximum time step per round, batch sampling size, discount factor, soft update coefficient, and exploration noise standard deviation.

[0060] S43. Interactive Sampling: Initialize the environment state s for each iteration. For each time step, each AGV... i Obtain local observations Select Action ; Execute joint operations Environmental feedback rewards and the next state ; Store the experience tuples to the replay pool; S44. Network Update: If the number of experiences in the replay pool is large, sample batch experiences for each AGV. i : Calculate the target value Update the Critic network: Minimize the loss Update the Actor network: Maximize performance Soft update target network: S45. Iteration Termination Judgment: Check if the preset number of iterations has been met. If the condition is not met, return to S43 and continue iterating; if the condition is met, output the trained Actor network parameters. S5, Energy Balance Decision Execution Module: The trained Actor network is deployed to the onboard controllers of each AGV. In actual operation: S51, Status Perception: AGV i The vehicle acquires its own status through onboard sensors and obtains the status of nearby charging stations and nearby AGVs through communication, forming a local observation system. S52, Action Decision: [This section appears to be incomplete and requires further context.] Input an Actor network, output actions Determine the starting and ending charging thresholds, then proceed to the selected charging station to charge.

[0061] S53. Energy balance is achieved: Each AGV executes the above decisions in a distributed manner. Through the equilibrium characteristics of potential game, a global energy balance state is spontaneously formed, that is, the power level of the AGV group is maintained at a uniform and high level, the charging station is fully utilized, and long-term idle time and AGV queuing congestion are avoided.

[0062] Therefore, this invention employs a potential game-theoretic deep reinforcement learning-driven AGV swarm energy balancing method, using the MADDPG-PGT multi-agent reinforcement learning algorithm to optimize the AGV swarm's power level, power distribution, and charging station resource utilization. A potential game model is introduced, and the existence of Nash equilibrium is rigorously proven to ensure algorithm convergence. A reward function based on potential function changes is designed to unify individual rationality and group energy efficiency. An Actor network parameter sharing mechanism is adopted to support flexible expansion of AGV swarms of different sizes. This method is applicable to different terminal scenarios such as vertical, U-shaped, and horizontal layouts, providing theoretical basis and management insights for the continuous high-intensity operation of automated terminals under dual carbon constraints.

[0063] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A charging decision-making method for AGV groups in an automated container terminal based on potential game deep reinforcement learning, characterized in that, Includes the following steps: Construct an energy balancing system model for an automated terminal AGV group; Based on the system model, an AGV group energy balance potential game model is established, which maps the charging decision behavior of each automated guided vehicle to a game strategy, and constructs a potential function consistent with the overall energy balance goal of the AGV group. Based on the aforementioned potential game model, a reward function with potential function changes as its core is designed; Based on the reward function, a potential game reinforcement learning algorithm framework is constructed, treating each automated guided vehicle as an independent decision-making agent. Based on local observation information, it outputs whether to charge and the corresponding charging control decision. In the training phase, a centralized value evaluation and distributed policy execution mechanism are introduced to jointly train the charging strategies of each automated guided vehicle. During the joint training process, the policy network and value evaluation network of each agent are iteratively updated through continuous interaction between each automated guided vehicle and the automated terminal simulation environment. After training, the policy networks of each agent are deployed to each automated guided vehicle (AGV). In actual operation, the AGVs autonomously make charging decisions based on their local states, thereby achieving distributed charging control and global energy balance for the AGV group under different dock layout conditions.

2. The method for AGV group charging decision-making in an automated container terminal based on potential game deep reinforcement learning as described in claim 1, characterized in that, The construction of the energy balance system model for the automated terminal AGV group specifically includes: An aggregated model is used to model the automated terminal operation system, defining an automated guided vehicle (AGV) set, a charging station set, a system time set, and a system operation state set. The AGV set represents multiple AGVs participating in charging decisions, the charging station set represents the charging facilities in the system that can charge AGVs, and the system operation state set describes the overall operation state of the automated terminal at any given time. Based on the aforementioned aggregated modeling, an energy balancing system model for an automated terminal AGV group is constructed. The remaining power status, operating position status, and task execution status of each automated guided vehicle at any given time, as well as the service occupancy status, queuing status, and spatial position status of each charging station at the corresponding time, are uniformly mapped to the global status of the facility, representing the comprehensive operating status of the system at that time. For the global state of the facility, a state reward function is defined to characterize the quality of system operation, expressed as: in, Indicates the global status of the facility The corresponding state reward value; This indicates the overall power level of the AGV group. This indicates the degree of dispersion in the power consumption of the AGV group. This indicates penalties for the use of charging station resources. , and These represent the weight coefficients of the corresponding items; Based on the aforementioned state-reward function, a charging strategy for the AGV group in any given group is defined. The individual benefit function for each automated guided vehicle is expressed as follows: in, Indicates the first Group charging strategy for automated guided vehicles Individual earnings value under the following conditions This refers to a group charging strategy comprised of the charging strategies of each automated guided vehicle (AGV). Indicates the first Charging strategy for automated guided vehicles (AGVs) Indicates the first A set of optional charging strategies for each automated guided vehicle; This indicates the charging strategy. The corresponding global state set of facilities.

3. The method for AGV group charging decision-making in an automated container terminal based on potential game deep reinforcement learning as described in claim 2, characterized in that, The overall power level of the AGV group , is represented as: in, Indicates the first Automated guided vehicles at time The remaining battery power, Indicates the number of automated guided vehicles (AGVs) participating in the charging decision-making process; The dispersion of the AGV group's power consumption , is represented as: The charging station resource utilization penalty item , is represented as: in, This indicates the penalty item corresponding to the charging station being idle. This indicates the penalty item corresponding to the queuing status at the charging station. These represent the weight coefficients of the charging station idle penalty item and the queuing penalty item, respectively.

4. The method for AGV group charging decision-making in an automated container terminal based on potential game deep reinforcement learning as described in claim 3, characterized in that, The penalty item corresponding to the idle state of the charging station , is represented as: in, Indicates the first A charging station at a time The service occupancy status is 1, which indicates that the charging station is in service, and 0 indicates that the charging station is idle. Indicates the number of charging stations; This represents the total statistical time of system operation. The penalty item corresponding to the queuing status at the charging station , is represented as: in, Indicates the first A charging station at a time The queue length, i.e., the charging station at any given time. Number of automated guided vehicles waiting to be charged.

5. The method for AGV group charging decision-making in an automated container terminal based on potential game deep reinforcement learning as described in claim 1, characterized in that, Based on the aforementioned system model, an AGV group energy balance potential game model is established, mapping the charging decision behavior of each automated guided vehicle to a game strategy, and constructing a potential function consistent with the overall energy balance objective of the AGV group, specifically including: The charging strategy of each automated guided vehicle is represented as an AGV group game strategy. ,in, Indicates the first Charging strategy for automated guided vehicles (AGVs) Indicates except the first Charging strategy for all automated guided vehicles except for the Automated Guided Vehicle (AGV); Based on the individual payoff functions of each automated guided vehicle (AGV) in the system model, an energy equilibrium potential game model for the AGV swarm is constructed, specifically including: In the aforementioned potential game model, a strategy update rule is set: when a strategy exists... Make Then the group charging strategy will be changed from Updated to ;in, This indicates a new group charging strategy; Indicates the first A new charging strategy for automated guided vehicles; Indicates the first Automated guided vehicles in the original group strategy Individual profit value; Indicates the first Automated guided vehicles in the updated swarm strategy Individual profit value; In the aforementioned potential game model, the Nash equilibrium condition is defined if and only if At that time, the set of strategies For the Nash equilibrium solution, where, Indicates the first A set of optional charging strategies for each automated guided vehicle.

6. The method for decision-making on charging of AGV groups in an automated container terminal based on potential game deep reinforcement learning as described in claim 1, characterized in that, The potential function is expressed as: in, Representing group policy The corresponding potential function value, This represents the total statistical time of system operation. Represents the global state set of the facility. Indicates at time Group strategy The corresponding state is in the global facility status. The number of AGVs, Indicates the global status of the facility The profit value.

7. The method for AGV group charging decision-making in an automated container terminal based on potential game deep reinforcement learning as described in claim 1, characterized in that, The reward function is defined as follows: in, Indicates the first Automated guided vehicles at time The reinforcement learning reward value, Indicates time Downstream charging strategy The corresponding potential function value, Indicates time Downstream charging strategy The corresponding potential function value.

8. The method for decision-making on charging of AGV groups in an automated container terminal based on potential game deep reinforcement learning as described in claim 1, characterized in that, The aforementioned reward function-based potential game reinforcement learning algorithm framework treats each automated guided vehicle (AGV) as an independent decision-making agent, outputting whether to charge and the corresponding charging control decision based on local observation information. During the training phase, a centralized value evaluation and distributed policy execution mechanism are introduced to jointly train the charging strategies of each AAV. Specifically, this includes: The AGV swarm is modeled as a multi-agent system, and the local observations of each automated guided vehicle are represented as follows: ,in, For the first The current battery level of the automated guided vehicle. For the first The current location of the automated guided vehicle. For the first The current mission status of the automated guided vehicle. For the first The target point of the automated guided vehicle. For the first Information on nearby charging stations for each automated guided vehicle includes location, queue length, and occupancy status. For the first Information about the surrounding AGVs of an automated guided vehicle, including their relative positions and task levels; At the same time, define the global state. ,in, This indicates the overall power distribution range of the AGV group. This represents the set of task levels for the AGV group. This is a set of AGV group position states. The first The status, queue length, and spatial location of each charging station. This refers to the set of automated guided vehicles (AGVs) that participate in the charging decision-making process. This represents the set of charging stations in the system; The actions of each automated guided vehicle are represented as a hybrid action space. Among them, discrete actions This indicates either to continue the task or to initiate a charging process. Indicates that charging has been initiated. Indicates continuation of the task; continuous actions. Used to select the initial charging threshold. and termination charging threshold ; The Actor-Critic framework is constructed based on a multi-agent deep deterministic policy gradient-potential game algorithm, with local observations input into the Actor network. Output action The Actor network adopts parameter sharing, meaning all automated guided vehicles (AGVs) share the same set of Actor network parameters; the Critic network inputs the AGV group charging strategy. s and joint actions Output potential function value This is used to evaluate the contribution of each AGV's actions to the group's energy balance. This indicates the number of automated guided vehicles (AGVs) involved in the charging decision-making process.

9. The method for decision-making on charging of AGV groups in an automated container terminal based on potential game deep reinforcement learning as described in claim 8, characterized in that, The loss function of the Critic network is expressed as: in, Indicates the first Parameters of the Critic network for automated guided vehicles. This represents the expectation for the experience replay sample. Indicates the parameters of the Critic network Below, group charging strategy and joint actions The corresponding potential function value, Indicates the first The target value for each automated guided vehicle is calculated using the following formula: in, Indicates the first Automated guided vehicles at time The reward value, This is a discount factor used to reflect the importance of future rewards. This represents the potential function value output by the Critic target network. This represents the action output by the Actor target network. Indicates the first Parameters of the Critic target network for an automated guided vehicle. This represents the updated set of group charging strategies; The Actor network parameters are updated via gradient ascent: in, Indicates the first Parameters of the Actor network for automated guided vehicles. This represents the objective function for the charging strategy performance. Indicates the first The gradient of the movement of an automated guided vehicle. This represents the potential function value output by the Critic network. This indicates that the Actor network is based on local observations. The output action; Indicates the first The gradient of the parameters of the Actor network for an automated guided vehicle.

10. The method for decision-making on charging of AGV groups in an automated container terminal based on potential game deep reinforcement learning as described in claim 1, characterized in that, The joint training specifically includes: During the training phase, the automated terminal simulation environment is used as a multi-agent interaction environment. The Actor network and Critic network of each automated guided vehicle are initialized, and an experience playback storage module is configured for each automated guided vehicle to store training sample data. During the interaction phase, for any given training moment, each automated guided vehicle (AGV) bases its training on the acquired local observations. Each actor network outputs its corresponding blended action. and in the mixed action By introducing exploratory noise, the combined actions of each automated guided vehicle (AGV) are applied to the automated terminal simulation environment to obtain an updated group charging strategy. and environmental status feedback; During the interaction phase, the reward value obtained by each automated guided vehicle at the current moment is calculated based on the reward function. and the mixed movements of all automated guided vehicles The current group charging strategy constitutes Local observation Mixed actions Reward Value and the updated group charging strategy Composition of empirical samples Stored in the experience replay pool; During the update phase, a batch of experience samples are randomly sampled from the experience replay pool. Based on the loss function and parameter update method, the Critic network parameters and Actor network parameters of each automated guided vehicle are updated. The Critic network adopts a centralized value evaluation method, and the Actor network adopts a distributed strategy update method.

Images