Port AGV dynamic battery replacement scheduling optimization method based on reinforcement learning

Abstract

The invention discloses a port AGV dynamic battery replacement scheduling optimization method based on reinforcement learning, and the method comprises the steps: S1, collecting the operation data of a port AGV, and constructing a training track according to a time step; s2, constructing environment state representation, and discretizing the station of the battery swap station and battery resources into a token set; s3, constructing an improved MAAC multi-agent decision model composed of an AGV strategy network and a centralized attention value network; s4, aggregating the AGV, the station and the battery token set, and calculating a joint value of candidate actions; s5, generating a feasible action set according to constraints such as reachable electric quantity and station capacity, and outputting optimal scheduling by a strategy network; s6, executing a scheduling action to update the resource states of the AGV and the battery swap station, and writing in a training track; and S7, based on the track centralized training model, outputting a scheduling sequence by the strategy network in a decentralized execution stage. According to the invention, the dynamism and the resource utilization rate of AGV battery replacement scheduling are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of port automation and reinforcement learning, and particularly relates to a port AGV dynamic battery replacement scheduling optimization method based on reinforcement learning. BACKGROUND

[0002] With the continuous improvement of port automation level, the automatic guided vehicle (AGV) has been widely used in container loading and unloading and horizontal transportation operations. In order to meet the demand of high-intensity and long-time continuous operation, the centralized battery replacement mode is generally used for energy supply of the port AGV, and the battery replacement scheduling problem brought by this mode gradually becomes a key factor affecting the port operation efficiency and system stability. In the actual port operating environment, the number of AGVs is large, the task changes frequently, the number of battery replacement station workstations and the battery inventory are limited, and there is a high coupling relationship between the AGV power state, the road network passing state and the task time limit, which puts high requirements on the real-time and collaboration of the battery replacement scheduling strategy.

[0003] In the prior art, the port AGV battery replacement scheduling method mainly relies on artificial experience rules, static priority strategies or centralized scheduling methods based on deterministic optimization models. Such methods are usually modeled under preset scenarios or simplified assumptions, and are difficult to adapt to the complex working conditions in the actual port, such as random task arrival, continuous change of AGV running state and dynamic occupation of battery replacement resources. When the system scale expands or the load fluctuates intensifies, problems such as AGV centralized queuing, uncontrollable battery replacement waiting time and individual AGVs forced to interrupt tasks due to insufficient power may occur. In addition, some research attempts to introduce single-agent or simple multi-agent reinforcement learning methods for scheduling decisions, but related methods often treat the battery replacement station state as an ordinary environmental variable, and fail to depict the competition relationship between multiple AGVs for key resources such as workstations and batteries, resulting in insufficient stability of the learning strategy in high-concurrency scenarios.

[0004] At the same time, the existing multi-agent reinforcement learning scheduling method often uses state splicing or average aggregation in the value evaluation process, which is difficult to distinguish the influence of different battery replacement decisions on specific battery replacement resources, and is prone to value estimation deviation, thereby affecting the rationality of the battery replacement decision. In addition, in the process of centralized training and decentralized execution, the existing method models the relationship between the battery replacement action and the resource state roughly, lacks a fine-grained description of the battery replacement queuing, workstation occupation time and battery inventory evolution process, resulting in that the scheduling strategy trained is difficult to stably reproduce the expected effect when actually deployed.

[0005] Therefore, how to provide a port AGV dynamic battery replacement scheduling optimization method based on reinforcement learning is a problem that those skilled in the art need to solve. SUMMARY

[0006] One purpose of the present application is to propose a port AGV dynamic battery replacement scheduling optimization method based on reinforcement learning, the present application constructs a port AGV dynamic battery replacement scheduling model based on an improved MAAC algorithm, by discretizing the battery replacement station and the battery resource into station tokens and battery tokens, participating in centralized attention value evaluation together with the local state of AGV, realizing fine modeling of the relationship between battery replacement scheduling action and resource occupation, and generating executable battery replacement scheduling decisions under the centralized training and decentralized execution mechanism combined with multiple physical constraints, so as to improve the stability and consistency of the scheduling results in the multi-AGV concurrent battery replacement scene.

[0007] According to the port AGV dynamic battery replacement scheduling optimization method based on reinforcement learning, the present application comprises the following steps: S1, collecting port AGV operation data and constructing training trajectories according to preset time steps; S2, constructing an environment state representation based on the port AGV operation data at each time step, and generating a station token set from the battery replacement station resource and a battery token set from the battery resource; S3, constructing a multi-agent decision model corresponding to the improved MAAC algorithm based on the environment state representation, the multi-agent decision model comprising a policy network corresponding to each AGV and a centralized attention value network; S4, in the improved MAAC algorithm, the centralized attention value network is used to aggregate the AGV set, the station token set and the battery token set, and the joint action value is calculated based on the candidate battery replacement action; S5, generating an actionable action set according to the power reach constraint, the road network access constraint, the station capacity constraint and the inventory availability constraint, and outputting a battery replacement scheduling action from the policy network in the actionable action set; S6, executing the battery replacement scheduling action to update the AGV state and the battery replacement station resource state, generating the next time step environment state and writing it into the training trajectory; S7, centralized training of the improved MAAC algorithm based on the training trajectory, and outputting a battery replacement scheduling action sequence from the policy network in the decentralized execution stage.

[0008] Optionally, the S1 comprises: S11, obtaining the original operation record of each AGV in the running process from the port automation control system, the original operation record comprising at least the position information, the power state information, the current task identification information, the task remaining distance information and the driving path information of each AGV at each sampling time; S12, obtain the battery swap station operation data aligned with the AGV operation record time from the port energy management system, the battery swap station operation data at least including the number of stations of each battery swap station, the occupation state of each station, the remaining service time corresponding to each station, the battery inventory quantity and the available state of each battery; S13, perform time synchronization processing on the AGV operation record and the battery swap station operation data, map the data of different sampling frequencies to a unified time axis, and discretize the continuous time axis in a preset time step to generate a continuous time step sequence; S14, taking two adjacent time steps as boundaries, segment the state change of each AGV in the continuous time step sequence, combine the segmented state sequence with the battery swap station operation data in the corresponding time step to form a time sequence sample containing the AGV state and the battery swap station resource state; S15, arrange the time sequence samples in chronological order, and associate the time sequence samples in continuous multiple time steps to form a training trajectory.

[0009] Optionally, the S2 includes: S21, read the data of the corresponding time step in the training trajectory at each time step, extract the current position, remaining power, current task identifier, task remaining distance and road section state of each AGV at the time step to form the local state information of the AGV; S22, extract the resource state information of each battery swap station at the time step, the resource state information including the battery swap station position, the total number of stations, the station occupation identifier, the remaining service time of each station, the battery inventory quantity and the available identifier of each battery; S23, discretize the station resource of the battery swap station into a plurality of station tokens according to the total number of stations in the battery swap station, and configure the station occupation identifier and the remaining service time attribute for each station token to form a station token set; S24, discretize the battery resource into a plurality of battery tokens according to the battery inventory quantity in the battery swap station, and configure the battery available identifier and the available time attribute for each battery token to form a battery token set; S25, combine the AGV local state information, the station token set and the battery token set according to the unified time step to generate the environment state representation of the corresponding time step.

[0010] Optionally, the S3 includes: S31, based on the environment state representation, construct an independent policy network for each AGV, the policy network taking the local state information corresponding to the AGV as input and outputting the battery swap scheduling action probability distribution corresponding to the AGV, and the policy networks do not share parameters; S32, a centralized attention value network is constructed, the centralized attention value network comprising a state encoding layer, an attention aggregation layer, and a joint value output layer, the state encoding layer respectively vectorizing AGV local state information, station token features, and battery token features; S33, in the attention aggregation layer, the encoded AGV state vector, station token vector, and battery token vector are mapped to a unified feature dimension, and a multi-head attention structure is constructed based on the unified feature dimension to perform parallel calculation on the correlation between different AGV state vectors, station token vectors, and battery token vectors; S34, an action embedding vector corresponding to each candidate battery replacement scheduling action is generated, and the action embedding vector is input into the multi-head attention structure as a query vector to perform correlation calculation with the AGV state vector, station token vector, and battery token vector respectively to obtain an attention weight distribution corresponding to the candidate battery replacement scheduling action; S35, the AGV state vector, station token vector, and battery token vector are weighted and converged based on the attention weight distribution, and the converged result is input into the joint value output layer together with the action embedding vector to generate a joint action value corresponding to the candidate battery replacement scheduling action.

[0011] Optionally, the S4 comprises: S41, an input vector set of the centralized attention value network is obtained at each time step, the input vector set comprising a local state vector set of each AGV, a station token vector set, a battery token vector set, and an action embedding vector set corresponding to each candidate battery replacement scheduling action; S42, linear mapping is performed on the action embedding vector corresponding to each candidate battery replacement scheduling action to generate a query vector, and linear mapping is performed on the AGV local state vector set, the station token vector set, and the battery token vector set to respectively generate an AGV key vector set and an AGV value vector set, a station token key vector set and a station token value vector set, and a battery token key vector set and a battery token value vector set; S43, correlation scores are calculated for the query vector and the corresponding key vector set one by one, an element-wise product is performed on the query vector and each key vector to obtain an original score, and the original score is divided by a preset scaling coefficient to obtain a scaled score, the preset scaling coefficient being the square root of the dimension of the query vector; S44. Normalize the AGV scaling score set, the workstation token scaling score set, and the battery token scaling score set respectively. Obtain the maximum value of all scaling scores in the corresponding set and perform subtraction on each scaling score to obtain the translation score. Perform exponential operation on each translation score to obtain the exponential value. Summate all exponential values ​​to obtain the normalization factor. Divide each exponential value by the normalization factor to obtain the attention weight. S45. Perform a weighted summation operation on the corresponding value vector set according to the attention weight to generate an aggregate vector. The aggregate vector includes an AGV aggregate vector, a workstation token aggregate vector, and a battery token aggregate vector. Then, concatenate the aggregate vector with the corresponding action embedding vector. The concatenation result is input into at least two fully connected network layers and output as a scalar in the last layer. The scalar is recorded as the joint action value corresponding to the candidate battery swapping scheduling action.

[0012] Optionally, S5 includes: S51. Read the environmental status at each time step, and extract the current position, power status, remaining distance of the current task, and passable road segment information associated with the current position of the AGV in the port road network. S52. Based on the road network distance between the current location of the AGV and the locations of each battery swapping station, and combined with the preset energy consumption parameters per unit distance, calculate the predicted energy consumption value required for the AGV to reach each battery swapping station, and mark the battery swapping station whose predicted energy consumption value is less than the current power of the AGV as a power reachable station. S53. For each power-accessible station, read the workstation token set and battery token set of the corresponding battery swapping station, filter workstation tokens with the workstation occupancy mark as unoccupied or with remaining service time less than a preset threshold, and filter battery tokens with the battery availability mark as available, and generate a candidate station set that meets the workstation capacity constraint and inventory availability constraint. S54. For each battery swapping station in the candidate station set, based on the remaining task distance of the AGV, the task deadline, and the remaining service time of the battery swapping station, determine the time feasibility of the AGV continuing to execute the current task after completing the battery swapping, and retain the battery swapping stations that meet the time constraints as the final selectable stations. S55. Based on the final selectable site, construct the set of possible actions for the AGV at the current time step.

[0013] Optionally, S6 includes: S61. Execute a battery swapping scheduling action at the current time step, determine the AGV's driving target position or the battery swapping station target position based on the battery swapping scheduling action, and update the AGV's position information. S62. Based on the AGV's travel distance in the current time step and the preset unit distance energy consumption parameter, calculate the AGV's energy consumption value in the time step, deduct the energy consumption value from the AGV's current power level, and update the AGV's power status. S63. When the AGV arrives at the battery swapping station and is assigned a corresponding work station token, the occupancy flag of the work station token is updated to an occupancy status, and the waiting time status of the AGV during the battery swapping process is updated according to the remaining service time of the work station token. S64. When the battery swap is completed, update the occupation flag of the workstation token to the unoccupied state, update the availability flag of the battery token to the unavailable state, and update the power status of the AGV to the rated power value of the corresponding battery. S65. After completing the location update, power update and battery swapping resource status update, re-summarize the status information of each AGV and the resource status information of each battery swapping station, generate the environmental status representation of the next time step, and associate the environmental status representation with the battery swapping scheduling action of the corresponding time step and write it into the training trajectory.

[0014] Optionally, S7 includes: S71. During the intensive training phase, read the training trajectory and extract the environmental state representation, battery swapping scheduling action, and environmental state representation of the next time step according to the time step sequence. S72. Based on the centralized attention value network, calculate the joint action value for each candidate battery swapping scheduling action in each time step, and form a time difference sample by combining the joint action value with the joint action value of adjacent time steps. S73. Perform gradient updates on the network parameters of the centralized attention value network based on the time difference samples; S74. Based on the joint action value output by the centralized attention value network, calculate the policy gradient for the policy network corresponding to each AGV, and independently update the network parameters of each policy network. S75. During the distributed execution phase, only the strategy network corresponding to each AGV is retained. Based on the local state information of the AGV at the current time step, the battery swapping scheduling action is output respectively, and the centralized attention value network is no longer called during the execution process.

[0015] The beneficial effects of this invention are: This invention constructs a dynamic battery swapping scheduling optimization framework for port AGVs based on an improved MAAC algorithm. In the multi-agent reinforcement learning modeling process, it unifies and structures the individual decisions of AGVs with key battery swapping resources such as station positions and battery inventory. By introducing station tokens and battery tokens during the environment state construction phase, it achieves discretized modeling of battery swapping resource occupancy status, service time, and availability, and combines this with the AGV's local operating state under the same time reference. In the improved MAAC algorithm, a centralized attention value network is used to perform action-conditional attention calculation on the AGV state vector, station token vector, and battery token vector. This allows the joint action value to distinguish the impact of different battery swapping scheduling actions on specific resource occupancy relationships during the evaluation process. Vector encoding, linear mapping, correlation calculation, scaling normalization, and weighted convergence operations are sequentially executed in the value calculation path to form a stable and consistent joint action value output. In the action generation phase, the set of possible actions is constrained and filtered by combining power availability constraints, road network access constraints, station capacity constraints, and inventory availability constraints. The policy network and value network are collaboratively updated under a centralized training and decentralized execution mechanism. Through the improved MAAC modeling and calculation process described above, this invention realizes refined coupling modeling between multi-AGV battery swapping decisions and battery swapping resource status, enabling the battery swapping scheduling process to have an implementable computational structure and stable decision output in a dynamic port environment, thereby improving the consistency of battery swapping scheduling results and the stability of system operation under the condition of concurrent operation of multiple AGVs. Attached Figure Description

[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0017] Fig. 1 This is a schematic diagram of the overall process of a dynamic battery swapping scheduling optimization method for port AGVs based on reinforcement learning proposed in this invention. Fig. 2 This is a schematic diagram of the multi-agent decision-making model structure of the improved MAAC algorithm in this invention; Fig. 3 This is a schematic diagram of the action-conditional attention calculation process of the centralized attention value network in this invention. Detailed Implementation

[0018] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0019] refer to Figs. 1-3 A dynamic battery swapping scheduling optimization method for port AGVs based on reinforcement learning includes the following steps: S1. Collect port AGV operation data and construct training trajectories according to preset time steps; S2. At each time step, construct an environmental status representation based on the port AGV operation data, and generate a set of workstation tokens for the battery swapping station workstation resources and a set of battery tokens for the battery resources. S3. Based on the environmental state representation, construct a multi-agent decision model corresponding to the improved MAAC algorithm. The multi-agent decision model includes a policy network and a centralized attention value network for each AGV. S4. In the improved MAAC algorithm, the centralized attention value network is used to aggregate the attention of the AGV set, the workstation token set, and the battery token set, and the joint action value is calculated based on the candidate battery swapping action. S5. Generate a set of actionable actions based on the power availability constraint, road network access constraint, workstation capacity constraint and inventory availability constraint, and output battery swapping scheduling actions within the set of actionable actions by the strategy network. S6. Execute the battery swapping scheduling action to update the AGV status and the battery swapping station resource status, generate the next time step environment status and write it into the training trajectory. S7. The improved MAAC algorithm is trained centrally based on the training trajectory, and the battery swapping scheduling action sequence is output by the network according to the strategy during the distributed execution phase.

[0020] In this embodiment, S1 includes: S11. Obtain the original operation records of each AGV during operation from the port automation control system. The original operation records shall include at least the position information, power status information, current task identifier information, remaining task distance information, and travel path information of each AGV at each sampling time. S12. Obtain the battery swapping station operation data aligned with the AGV operation record time from the port energy management system. The battery swapping station operation data includes at least the number of workstations in each battery swapping station, the occupancy status of each workstation, the remaining service time corresponding to each workstation, the battery inventory quantity, and the availability status of each battery. S13. Perform time synchronization processing on the AGV operation record and the battery swapping station operation data, map data with different sampling frequencies to a unified time axis, and discretize the continuous time axis with a preset time step to generate a continuous time step sequence. S14. Using two adjacent time steps as boundaries, the state changes of each AGV in the continuous time step sequence are segmented, and the segmented state sequence is combined with the operation data of the battery swapping station in the corresponding time step to form a time series sample containing the AGV state and the battery swapping station resource state. S15. Arrange the time series samples in chronological order, and associate the time series samples in multiple consecutive time steps to form a training trajectory.

[0021] In this embodiment, S2 includes: S21. Read the data of the corresponding time step in the training trajectory at each time step, and extract the current position, remaining power, current task identifier, remaining task distance and road segment status of each AGV at the time step to form the local state information of the AGV. S22. Extract the resource status information of each battery swapping station at the time step. The resource status information includes the location of the battery swapping station, the total number of workstations, the occupation identifier of each workstation, the remaining service time of each workstation, the battery inventory quantity, and the available identifier of each battery. S23. Based on the total number of workstations in the swapping station, the workstation resources of the swapping station are discretized into multiple workstation tokens. For each workstation token, a workstation occupancy identifier and remaining service time attribute are configured to form a set of workstation tokens. S24. Based on the battery inventory quantity in the battery swapping station, the battery resources are discretized into multiple battery tokens. For each battery token, a battery availability identifier and availability time attribute are configured to form a battery token set. S25. Combine the AGV local state information, the workstation token set, and the battery token set according to a unified time step to generate an environmental state representation corresponding to the time step.

[0022] In this invention, workstation tokens and battery tokens are not merely used as status identifiers, but rather as independent discrete resource units participating in subsequent multi-agent decision-making modeling. Each workstation token and battery token is uniquely associated with a specific battery swapping station upon generation, and its corresponding occupancy status, remaining service time, and availability are updated with each time step, enabling battery swapping resources to form a traceable resource evolution sequence in the temporal dimension. By discretizing the continuously changing battery swapping resource status into multiple token objects, the environmental state maintains temporal consistency while possessing a fixed structural expression, thus allowing subsequent attention modeling to reuse the same network structure across battery swapping stations of different sizes, ensuring stable training and deployability of the model in multi-site, multi-workstation scenarios.

[0023] In this embodiment, S3 includes: S31. Based on the environmental state representation, an independent strategy network is constructed for each AGV. The strategy network takes the local state information corresponding to the AGV as input and outputs the probability distribution of battery swapping scheduling actions corresponding to the AGV. The strategy networks do not share parameters. S32. Construct a centralized attention value network, which includes a state encoding layer, an attention aggregation layer and a joint value output layer. The state encoding layer performs vectorized encoding on the AGV local state information, workstation token features and battery token features, respectively. S33. In the attention aggregation layer, the encoded AGV state vector, workstation token vector and battery token vector are mapped to a unified feature dimension, and a multi-head attention structure is constructed based on the unified feature dimension to perform parallel calculation on the correlation between different AGV state vectors, workstation token vectors and battery token vectors. S34. Generate a corresponding action embedding vector for each candidate battery swapping scheduling action, and input the action embedding vector into the multi-head attention structure as a query vector. Perform correlation calculation with the AGV state vector, workstation token vector and battery token vector respectively to obtain the attention weight distribution corresponding to the candidate battery swapping scheduling action. S35. Based on the attention weight distribution, the AGV state vector, workstation token vector, and battery token vector are weighted and converged, and the convergence result and the action embedding vector are input into the joint value output layer to generate the joint action value of the corresponding candidate battery swapping scheduling action.

[0024] In the centralized attention value network, the AGV local state vector, workstation token vector, and battery token vector are arranged with a unified vector dimension and encoding order before entering the attention aggregation layer. They are then computed in parallel according to the head index within the multi-head attention structure, with each attention head corresponding to independent linear mapping parameters. Action embedding vectors generated for different candidate battery swapping scheduling actions participate in correlation calculations with the resource token vector and AGV state vector during the attention calculation process, and each candidate action corresponds to an independent set of attention weight distributions. This structure enables the joint action value to be calculated within the network using a "action-resource-state" ternary relationship, ensuring a consistent computational flow and stable parameter scale for centralized value assessment in scenarios with multiple battery swapping stations and multiple workstations.

[0025] In this embodiment, S4 includes: S41. At each time step, obtain the input vector set of the centralized attention value network. The input vector set includes the local state vector set of each AGV, the workstation token vector set, the battery token vector set, and the action embedding vector set corresponding to each candidate battery swapping scheduling action. S42. Perform linear mapping on the action embedding vector corresponding to each candidate battery swapping scheduling action to generate a query vector, and perform linear mapping on the AGV local state vector set, the workstation token vector set, and the battery token vector set to generate the AGV key vector set and AGV value vector set, the workstation token key vector set and workstation token value vector set, and the battery token key vector set and battery token value vector set, respectively. S43. Calculate the relevance score for each query vector and the corresponding key vector set, perform element-wise multiplication on the query vector and each key vector, sum the product results to obtain the original score, and divide the original score by a preset scaling factor to obtain the scaling score. The preset scaling factor is the square root of the query vector dimension. S44. Normalize the AGV scaling score set, the workstation token scaling score set, and the battery token scaling score set respectively. Obtain the maximum value of all scaling scores in the corresponding set and perform subtraction on each scaling score to obtain the translation score. Perform exponential operation on each translation score to obtain the exponential value. Summate all exponential values ​​to obtain the normalization factor. Divide each exponential value by the normalization factor to obtain the attention weight. S45. Perform a weighted summation operation on the corresponding value vector set according to the attention weight to generate an aggregate vector. The aggregate vector includes an AGV aggregate vector, a workstation token aggregate vector, and a battery token aggregate vector. Then, concatenate the aggregate vector with the corresponding action embedding vector. The concatenation result is input into at least two fully connected network layers and output as a scalar in the last layer. The scalar is recorded as the joint action value corresponding to the candidate battery swapping scheduling action.

[0026] In the joint action value calculation process, for the same candidate battery swapping scheduling action, the AGV key-value vector set, the workstation token key-value vector set, and the battery token key-value vector set participate in independent correlation calculation and normalization processing, respectively. The attention weights corresponding to each set are not mixed across sets during the calculation process. Various aggregated vectors are concatenated in a preset order after generation. This preset order is fixed during the network initialization phase and remains consistent during subsequent training and inference. The number of layers, the number of neurons in each layer, and the activation method of the at least two fully connected networks are determined during the model construction phase and remain unchanged at different time steps, thereby ensuring that the joint action value has a consistent calculation path and stable parameter structure under different candidate actions and different resource scales.

[0027] In this embodiment, S5 includes: S51. Read the environmental status at each time step, and extract the current position, power status, remaining distance of the current task, and passable road segment information associated with the current position of the AGV in the port road network. S52. Based on the road network distance between the current location of the AGV and the locations of each battery swapping station, and combined with the preset energy consumption parameters per unit distance, calculate the predicted energy consumption value required for the AGV to reach each battery swapping station, and mark the battery swapping station whose predicted energy consumption value is less than the current power of the AGV as a power reachable station. S53. For each power-accessible station, read the workstation token set and battery token set of the corresponding battery swapping station, filter workstation tokens with the workstation occupancy mark as unoccupied or with remaining service time less than a preset threshold, and filter battery tokens with the battery availability mark as available, and generate a candidate station set that meets the workstation capacity constraint and inventory availability constraint. S54. For each battery swapping station in the candidate station set, based on the remaining task distance of the AGV, the task deadline, and the remaining service time of the battery swapping station, determine the time feasibility of the AGV continuing to execute the current task after completing the battery swapping, and retain the battery swapping stations that meet the time constraints as the final selectable stations. S55. Based on the final selectable site, construct the set of possible actions for the AGV at the current time step.

[0028] In the process of calculating the predicted energy consumption, the road network distance is obtained by accumulating the sequence of road segments from the current position of the AGV to the location of the battery swapping station. The energy consumption parameter per unit distance is pre-configured according to the AGV model and stored separately for different AGVs. The task deadline and the remaining service time of the battery swapping station are compared under the same time reference. The battery swapping completion time is obtained by adding the arrival travel time to the corresponding workstation's remaining service time. The final set of selectable stations generated for battery swapping stations that meet all constraints remains unchanged within the current time step and serves as a fixed input for subsequent action outputs and state updates, thereby ensuring the determinism and consistency of the set of actionable actions within a single decision cycle.

[0029] In this embodiment, S6 includes: S61. Execute a battery swapping scheduling action at the current time step, determine the AGV's driving target position or the battery swapping station target position based on the battery swapping scheduling action, and update the AGV's position information. S62. Based on the AGV's travel distance in the current time step and the preset unit distance energy consumption parameter, calculate the AGV's energy consumption value in the time step, deduct the energy consumption value from the AGV's current power level, and update the AGV's power status. S63. When the AGV arrives at the battery swapping station and is assigned a corresponding work station token, the occupancy flag of the work station token is updated to an occupancy status, and the waiting time status of the AGV during the battery swapping process is updated according to the remaining service time of the work station token. S64. When the battery swap is completed, update the occupation flag of the workstation token to the unoccupied state, update the availability flag of the battery token to the unavailable state, and update the power status of the AGV to the rated power value of the corresponding battery. S65. After completing the location update, power update and battery swapping resource status update, re-summarize the status information of each AGV and the resource status information of each battery swapping station, generate the environmental status representation of the next time step, and associate the environmental status representation with the battery swapping scheduling action of the corresponding time step and write it into the training trajectory.

[0030] After the battery swapping scheduling action is executed, the AGV location information is updated segment by segment based on the connection relationship between adjacent road segments in the road network. The travel distance is obtained by accumulating the actual length of the road segments traversed within the current time step. The status updates of the workstation token and battery token are strictly synchronized with the time step. The remaining service time of the workstation token decreases at a fixed time granularity within each time step, triggering a battery swapping completion event when it decreases to zero. The data of each time step in the training trajectory is stored in chronological order as a combination record of state, action, and resource state, and the time step length remains consistent during trajectory generation to ensure the complete reproduction of the state evolution process in the subsequent intensive training phase.

[0031] In this embodiment, S7 includes: S71. During the intensive training phase, read the training trajectory and extract the environmental state representation, battery swapping scheduling action, and environmental state representation of the next time step according to the time step sequence. S72. Based on the centralized attention value network, calculate the joint action value for each candidate battery swapping scheduling action in each time step, and form a time difference sample by combining the joint action value with the joint action value of adjacent time steps. S73. Perform gradient updates on the network parameters of the centralized attention value network based on the time difference samples; S74. Based on the joint action value output by the centralized attention value network, calculate the policy gradient for the policy network corresponding to each AGV, and independently update the network parameters of each policy network. S75. During the distributed execution phase, only the strategy network corresponding to each AGV is retained. Based on the local state information of the AGV at the current time step, the battery swapping scheduling action is output respectively, and the centralized attention value network is no longer called during the execution process.

[0032] During the centralized training phase, the centralized attention value network and the policy networks of each AGV are updated synchronously within the same training cycle. Training batches are sampled sequentially according to time steps to maintain consistency in state transition relationships. The joint action values ​​involved in the time difference samples are weighted and combined with adjacent time steps according to a fixed discount factor, which is set during model initialization and remains unchanged during training. During the decentralized execution phase, each AGV only loads its own corresponding policy network parameters. The input dimension of the policy network is consistent with the local state information dimension, thereby ensuring structural consistency in state representation between the training and execution phases and ensuring that scheduling decisions can be stably reproduced according to the training results during actual deployment.

[0033] Example 1: To verify the feasibility and effectiveness of this invention in a real port operation environment, the proposed dynamic battery swapping scheduling optimization method for port AGVs based on the improved MAAC algorithm was applied to the AGV transportation and battery swapping scheduling scenario of a large automated container terminal. This port is a typical fully automated terminal, where AGVs are used for container transfer between quay cranes, yard cranes, and the horizontal transport system. The AGVs are recharged using a centralized battery swapping method, with battery swapping stations distributed at the edge of the yard and at nodes of the main road network.

[0034] In this implementation scenario, the port deploys 48 AGVs, each equipped with a uniform type of power battery with a rated capacity of 120 kWh, capable of supporting approximately 90 minutes of continuous operation on a single full charge. The port has four battery swapping stations, each with three swapping bays, and a battery inventory of 6 to 10 sets. The average swapping service time is 6 minutes. Port operations exhibit significant dynamic characteristics; transport tasks generated by quay crane loading and unloading arrive randomly. AGVs face challenges such as continuous battery consumption, path congestion changes, and competition for battery swapping resources during task execution. Before the introduction of this invention, the port primarily employed a rule-based scheduling strategy based on a fixed battery threshold and proximity principle. When an AGV's battery level fell below a preset threshold, it would directly proceed to the nearest battery swapping station. This strategy, under high load conditions, frequently resulted in multiple AGVs arriving at the same swapping station simultaneously, significantly increasing queuing times and causing some AGVs to experience prolonged waiting periods or even task interruptions.

[0035] In this embodiment, the operation data of the port AGV and the battery swapping station are first collected and processed. The collected data includes the real-time location of the AGV, its battery status, remaining distance of the task, task deadline, travel path information, and the occupancy status and battery inventory status of the battery swapping station. Through time synchronization processing, data from different sources and with different sampling frequencies are mapped to a unified time axis, and a training trajectory is constructed according to a 30-second time step. During the environmental state construction process, the battery swapping station resources are discretized into station tokens, and the battery inventory resources are discretized into battery tokens, which, together with the local state of the AGV, form a structured environmental state representation.

[0036] In the scheduling model construction phase, the improved MAAC algorithm proposed in this invention is used to build an independent policy network for each AGV and a centralized attention value network. In the centralized attention value network, the AGV state vector, station token vector, and battery token vector participate in multi-head attention calculation after being mapped through a unified dimension. Action embedding vectors are introduced for different candidate battery swapping scheduling actions, enabling the calculation process of joint action value to reflect the impact of different battery swapping decisions on the specific battery swapping resource occupancy relationship. In the action generation phase, the set of reachable battery swapping stations is calculated by combining the AGV's current power level and road network distance. Furthermore, the final set of actionable actions is filtered based on station occupancy status, battery availability status, and task time constraints, ensuring that the battery swapping scheduling actions output by the policy network are executable at both the physical and temporal levels.

[0037] During training, a centralized training and decentralized execution mechanism was adopted. The training trajectory constructed from historical operational data was input into the improved MAAC model for training. The training period covered 14 consecutive days of port operation data, including various operating conditions such as high load and low load. After training, only the strategy network corresponding to each AGV was retained in actual operation. Based on the local state of the AGV, the battery swapping scheduling decision was output in real time, and the centralized value network was no longer invoked, thus meeting the real-time scheduling requirements.

[0038] To verify the beneficial effects of the present invention, the scheduling results using the method of the present invention were compared with the original rule-based scheduling strategy, and key operational indicators were statistically analyzed during a continuous 7-day operation. The experimental results are shown in Table 1: Table 1. Comparison of Port AGV Operation Indicators under Different Battery Swapping Dispatch Methods

[0039] As shown in Table 1, the method of this invention exhibits significant advantages in several key operational indicators. After adopting the improved MAAC algorithm, the average waiting time for a single battery swap by the AGV decreased from 18.6 minutes to 9.4 minutes, a reduction of approximately 49.5%, and the maximum battery swap queuing time decreased from 41.2 minutes to 17.8 minutes, indicating a more balanced scheduling of battery swapping resources and effective alleviation of queuing congestion. The number of AGV task interruptions due to insufficient power decreased from 6.3 times per day to 0.8 times, demonstrating that battery swapping decisions can be made before power risks occur. Simultaneously, the average utilization rate of battery swapping station workstations increased from 58.1% to 76.4%, achieving improved resource utilization efficiency without increasing hardware resources. The average daily transport volume completed by the AGV increased by approximately 14.5%, and the number of battery swapping congestion occurrences and the volatility of system scheduling decisions during peak hours both decreased significantly, further verifying the effectiveness of this invention in improving system stability and scheduling consistency under high load conditions. The improved MAAC dynamic battery swapping scheduling method proposed in this invention can effectively solve problems such as intense competition for battery swapping resources, uncontrollable queuing time, and unstable scheduling results under the condition of multiple AGVs operating concurrently in real port operation scenarios. It has good engineering feasibility and significant practical application value.

[0040] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A dynamic battery swapping scheduling optimization method for port AGVs based on reinforcement learning, characterized in that, Includes the following steps: S1. Collect port AGV operation data and construct training trajectories according to preset time steps; S2. At each time step, construct an environmental status representation based on the port AGV operation data, and generate a set of workstation tokens for the battery swapping station workstation resources and a set of battery tokens for the battery resources. S3. Based on the environmental state representation, construct a multi-agent decision model corresponding to the improved MAAC algorithm. The multi-agent decision model includes a policy network and a centralized attention value network for each AGV. S4. In the improved MAAC algorithm, the centralized attention value network is used to aggregate the attention of the AGV set, the workstation token set, and the battery token set, and the joint action value is calculated based on the candidate battery swapping action. S5. Generate a set of actionable actions based on the power availability constraint, road network access constraint, workstation capacity constraint and inventory availability constraint, and output battery swapping scheduling actions within the set of actionable actions by the strategy network. S6. Execute the battery swapping scheduling action to update the AGV status and the battery swapping station resource status, generate the next time step environment status and write it into the training trajectory. S7. The improved MAAC algorithm is trained centrally based on the training trajectory, and the battery swapping scheduling action sequence is output by the network according to the strategy during the distributed execution phase.

2. The method for dynamic battery swapping scheduling optimization of port AGVs based on reinforcement learning according to claim 1, characterized in that, S1 includes: S11. Obtain the original operation records of each AGV during operation from the port automation control system. The original operation records shall include at least the position information, power status information, current task identifier information, remaining task distance information, and travel path information of each AGV at each sampling time. S12. Obtain the battery swapping station operation data aligned with the AGV operation record time from the port energy management system. The battery swapping station operation data includes at least the number of workstations in each battery swapping station, the occupancy status of each workstation, the remaining service time corresponding to each workstation, the battery inventory quantity, and the availability status of each battery. S13. Perform time synchronization processing on the AGV operation record and the battery swapping station operation data, map data with different sampling frequencies to a unified time axis, and discretize the continuous time axis with a preset time step to generate a continuous time step sequence. S14. Using two adjacent time steps as boundaries, the state changes of each AGV in the continuous time step sequence are segmented, and the segmented state sequence is combined with the operation data of the battery swapping station in the corresponding time step to form a time series sample containing the AGV state and the battery swapping station resource state. S15. Arrange the time series samples in chronological order, and associate the time series samples in multiple consecutive time steps to form a training trajectory.

3. The method for dynamic battery swapping scheduling optimization of port AGVs based on reinforcement learning according to claim 1, characterized in that, S2 includes: S21. Read the data of the corresponding time step in the training trajectory at each time step, and extract the current position, remaining power, current task identifier, remaining task distance and road segment status of each AGV at the time step to form the local state information of the AGV. S22. Extract the resource status information of each battery swapping station at the time step. The resource status information includes the location of the battery swapping station, the total number of workstations, the occupation identifier of each workstation, the remaining service time of each workstation, the battery inventory quantity, and the available identifier of each battery. S23. Based on the total number of workstations in the swapping station, the workstation resources of the swapping station are discretized into multiple workstation tokens. For each workstation token, a workstation occupancy identifier and remaining service time attribute are configured to form a set of workstation tokens. S24. Based on the battery inventory quantity in the battery swapping station, the battery resources are discretized into multiple battery tokens. For each battery token, a battery availability identifier and availability time attribute are configured to form a battery token set. S25. Combine the AGV local state information, the workstation token set, and the battery token set according to a unified time step to generate an environmental state representation corresponding to the time step.

4. The method for dynamic battery swapping scheduling optimization of port AGVs based on reinforcement learning according to claim 1, characterized in that, S3 includes: S31. Based on the environmental state representation, an independent strategy network is constructed for each AGV. The strategy network takes the local state information corresponding to the AGV as input and outputs the probability distribution of battery swapping scheduling actions corresponding to the AGV. The strategy networks do not share parameters. S32. Construct a centralized attention value network, which includes a state encoding layer, an attention aggregation layer and a joint value output layer. The state encoding layer performs vectorized encoding on the AGV local state information, workstation token features and battery token features, respectively. S33. In the attention aggregation layer, the encoded AGV state vector, workstation token vector and battery token vector are mapped to a unified feature dimension, and a multi-head attention structure is constructed based on the unified feature dimension to perform parallel calculation on the correlation between different AGV state vectors, workstation token vectors and battery token vectors. S34. Generate a corresponding action embedding vector for each candidate battery swapping scheduling action, and input the action embedding vector into the multi-head attention structure as a query vector. Perform correlation calculation with the AGV state vector, workstation token vector and battery token vector respectively to obtain the attention weight distribution corresponding to the candidate battery swapping scheduling action. S35. Based on the attention weight distribution, the AGV state vector, workstation token vector, and battery token vector are weighted and converged, and the convergence result and the action embedding vector are input into the joint value output layer to generate the joint action value of the corresponding candidate battery swapping scheduling action.

5. The method for dynamic battery swapping scheduling optimization of port AGVs based on reinforcement learning according to claim 1, characterized in that, S4 includes: S41. At each time step, obtain the input vector set of the centralized attention value network. The input vector set includes the local state vector set of each AGV, the workstation token vector set, the battery token vector set, and the action embedding vector set corresponding to each candidate battery swapping scheduling action. S42. Perform linear mapping on the action embedding vector corresponding to each candidate battery swapping scheduling action to generate a query vector, and perform linear mapping on the AGV local state vector set, the workstation token vector set, and the battery token vector set to generate the AGV key vector set and AGV value vector set, the workstation token key vector set and workstation token value vector set, and the battery token key vector set and battery token value vector set, respectively. S43. Calculate the relevance score for each query vector and the corresponding key vector set, perform element-wise multiplication on the query vector and each key vector, sum the product results to obtain the original score, and divide the original score by a preset scaling factor to obtain the scaling score. The preset scaling factor is the square root of the query vector dimension. S44. Normalize the AGV scaling score set, the workstation token scaling score set, and the battery token scaling score set respectively. Obtain the maximum value of all scaling scores in the corresponding set and perform subtraction on each scaling score to obtain the translation score. Perform exponential operation on each translation score to obtain the exponential value. Summate all exponential values ​​to obtain the normalization factor. Divide each exponential value by the normalization factor to obtain the attention weight. S45. Perform a weighted summation operation on the corresponding value vector set according to the attention weight to generate an aggregate vector. The aggregate vector includes an AGV aggregate vector, a workstation token aggregate vector, and a battery token aggregate vector. Then, concatenate the aggregate vector with the corresponding action embedding vector. The concatenation result is input into at least two fully connected network layers and output as a scalar in the last layer. The scalar is recorded as the joint action value corresponding to the candidate battery swapping scheduling action.

6. The method for dynamic battery swapping scheduling optimization of port AGVs based on reinforcement learning according to claim 1, characterized in that, S5 includes: S51. Read the environmental status at each time step, and extract the current position, power status, remaining distance of the current task, and passable road segment information associated with the current position of the AGV in the port road network. S52. Based on the road network distance between the current location of the AGV and the locations of each battery swapping station, and combined with the preset energy consumption parameters per unit distance, calculate the predicted energy consumption value required for the AGV to reach each battery swapping station, and mark the battery swapping station whose predicted energy consumption value is less than the current power of the AGV as a power reachable station. S53. For each power-accessible station, read the workstation token set and battery token set of the corresponding battery swapping station, filter workstation tokens with the workstation occupancy mark as unoccupied or with remaining service time less than a preset threshold, and filter battery tokens with the battery availability mark as available, and generate a candidate station set that meets the workstation capacity constraint and inventory availability constraint. S54. For each battery swapping station in the candidate station set, based on the remaining task distance of the AGV, the task deadline, and the remaining service time of the battery swapping station, determine the time feasibility of the AGV continuing to execute the current task after completing the battery swapping, and retain the battery swapping stations that meet the time constraints as the final selectable stations. S55. Based on the final selectable site, construct the set of possible actions for the AGV at the current time step.

7. The method for dynamic battery swapping scheduling optimization of port AGVs based on reinforcement learning according to claim 1, characterized in that, S6 includes: S61. Execute a battery swapping scheduling action at the current time step, determine the AGV's driving target position or the battery swapping station target position based on the battery swapping scheduling action, and update the AGV's position information. S62. Based on the AGV's travel distance in the current time step and the preset unit distance energy consumption parameter, calculate the AGV's energy consumption value in the time step, deduct the energy consumption value from the AGV's current power level, and update the AGV's power status. S63. When the AGV arrives at the battery swapping station and is assigned a corresponding work station token, the occupancy flag of the work station token is updated to an occupancy status, and the waiting time status of the AGV during the battery swapping process is updated according to the remaining service time of the work station token. S64. When the battery swap is completed, update the occupation flag of the workstation token to the unoccupied state, update the availability flag of the battery token to the unavailable state, and update the power status of the AGV to the rated power value of the corresponding battery. S65. After completing the location update, power update and battery swapping resource status update, re-summarize the status information of each AGV and the resource status information of each battery swapping station, generate the environmental status representation of the next time step, and associate the environmental status representation with the battery swapping scheduling action of the corresponding time step and write it into the training trajectory.

8. The method for dynamic battery swapping scheduling optimization of port AGVs based on reinforcement learning according to claim 1, characterized in that, S7 includes: S71. During the intensive training phase, read the training trajectory and extract the environmental state representation, battery swapping scheduling action, and environmental state representation of the next time step according to the time step sequence. S72. Based on the centralized attention value network, calculate the joint action value for each candidate battery swapping scheduling action in each time step, and form a time difference sample by combining the joint action value with the joint action value of adjacent time steps. S73. Perform gradient updates on the network parameters of the centralized attention value network based on the time difference samples; S74. Based on the joint action value output by the centralized attention value network, calculate the policy gradient for the policy network corresponding to each AGV, and independently update the network parameters of each policy network. S75. During the distributed execution phase, only the strategy network corresponding to each AGV is retained. Based on the local state information of the AGV at the current time step, the battery swapping scheduling action is output respectively, and the centralized attention value network is no longer called during the execution process.

Images